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Abstract 

MultiChord is a new variant of the Chord namespace management algorithm [7] that includes lightweight mech- 
anisms for accommodating a limited rate of change, specifically, process joins and failures. This paper describes the 
algorithm formally and evaluates its performance, using both simulation and analysis. Our main result is that lookups 
are provably correct — that is, each lookup returns results that are consistent with a hypothetical ideal system that dif- 
fers from the actual system only in entries corresponding to recent joins and failures — in the presence of a limited rate 
of change. In particular, if the number of joins and failures that occur during a given time interval in a given region 
of system are bounded, then all lookups are correct. A second result is a guaranteed upper bound for the latency of a 
lookup operation in the absence of any other lookups in the system. Finally, we establish a relationship between the 
deterministic assumptions of bounded joins and failures and the probabilistic assumptions (which are often used to 
model large scale networks). In particular, we derive a lower bound for the mean time between two violations of the 
deterministic assumptions in a steady state system where joins and failures are modeled by Poisson processes. 



1 Introduction 

This paper describes MultiChord, a new, more resilient variant of the Chord namespace management algorithm [7]. 
The main innovation is that MultiChord includes lightweight mechanisms for accommodating a limited rate of change, 
specifically, process joins and failures. 

The contributions of this paper include (a) techniques for improving the performance and resiliency of peer-to-peer 
namespace management algorithms, and (b) methods of analyzing performance for such algorithms in the presence of 
a bounded rate of change. 

Building in resiliency: We improve the performance and resiliency of Chord by adding additional entries to processes' 
routing (finger) tables, and by delaying a process from joining until its finger table is properly populated. This demon- 
strates an approach to building peer-to-peer namespace management services in which resiliency to a bounded rate 
of change is built in from the beginning. The method we use is to design the ideal communication infrastructure 
with enough redundancy to accommodate a bounded rate of change without reducing latency, and to maintain this 
redundant structure using gossiping. Newly-joining processes should not participate fully in the system until they 
have been fully incorporated into the communication infrastructure. This general approach should extend to other 
communication infrastructures such as those proposed in [1, 4, 5, 6, 7]. 

Formal modeling and analysis: We present the algorithm precisely, using high-level, nondeterministic timed I/O 
automata pseudocode. We analyze its performance conditionally, assuming a limited rate of change. This demonstrates 
how peer-to-peer namespace management algorithms can be modeled using state machines and subjected to proofs 
and analysis. In particular, it demonstrates that interesting performance results can be obtained for such algorithms 
using conditional analysis, conditioned on the "normal case" assumption that changes happen at a bounded rate. This 
kind of analysis should be useful in comparing different namespace management algorithms. 

Our method of analysis is quite different from the probabilistic style used by Liben-Nowell et al [2]. Our claims 
are not probabilistic, but rather, worst-case bounds under restricted circumstances. Our assumptions about the rate of 
change are rather strong. However, as we discuss in Section 3, we can relax these assumptions by adding probabilistic 
assumptions, while still obtaining our stronger latency bounds. 

1.1 Overview 

The original Chord protocol [7] assumes a circular identifier space (called the Chord ring) of size N = 2™. With each 
process i is associated a unique logical identifier in this space. Each process i maintains a routing table (known as a 
finger table). The A;-th entry in this table, called the k-th finger of process i, contains a reference to the first process 
whose logical identifier follows process i's logical identifier by at least 2 k in the clockwise direction on the Chord 
ring, where < k < n. In the remainder of this paper we refer to these fingers as the power-of-two fingers of i. 
The successor of a logical identifier id represents the first process whose logical identifier follows id in the clockwise 
direction on the Chord ring, or the process with logical identifier id if such a process exists. We redefine the notion of 
successor in the context of MultiChord in Section 1.2. 

In MultiChord, process i maintains, in addition to the finger table like that used in Chord, information about its 
"6-block" (i.e., its own b successors and b predecessors) and all 6-blocks of its power-of-two fingers. The value of b 
is chosen based on an assumed upper bound on the "normal" rate of change. When the algorithm is in an ideal state, 
each process' finger table contains its 6-block, as well as a 6-block for each of its power-of-two fingers. However, this 
information can degrade from an ideal state as a result of process joins and failures. 

MultiChord includes lightweight mechanisms, based on periodic background gossiping, for maintaining the system 
in a nearly ideal state in the face of limited change, i.e., limited joins and failures. Each process i continually sends 
its own 6-block to its b successors and b predecessors, which allows them to update their finger tables. In addition, 
process i continually "pings" its power-of-two fingers, who respond by returning their own 6-blocks. These periodic 
exchanges of information between a process and the processes in its finger table allow the system to gravitate back 
toward an ideal state in the face of changes. Like Chord, MultiChord does not differentiate between a process failure 
and departure. When a process i fails or leaves, processes who maintain process i in their finger tables will remove it 
when it expires. 

When a new process i joins the system, it first populates its finger table with its 6-block, and the 6-blocks of its 
power-of-two fingers. Like in Chord, a process i uses the lookup operation to find its power-of-two fingers. There 



are two other instances when a process i invokes a lookup: (i) when a client at location i explicitly invokes a lookup 
operation for a specified target, and (iii) when it decides to refresh its finger table. 

Like Chord, MultiChord implements the lookup operation in an iterative fashion. Consider a process i that per- 
forms a lookup on value x. At every iteration (stage), process i sends a query to the best known predecessor for x. Let 
process k be this predecessor. Upon receiving the query, process k checks whether it knows the process responsible 
for rr-that is, whether its immediate successor is responsible for x — , and if yes, it sends the answer back to process i. 
Otherwise process k sends its best known predecessor for x to i. MultiChord generalizes this procedure: at every stage 
process i sends c > 1 queries to the best known c predecessors for x. In turn, process k responds with its best known 
c predecessors of x. As we will show this redundancy increases the resilience of the lookup in the face of changes. 

The value of c is chosen to be larger than the number of changes that "normally" occur in a "small" interval of time, 
in a limited region of the ring. The length of this small interval of time is assumed to be sufficient for the system to 
recover from a limited number of changes in the relevant region of the ring. The admissible rate of change is quantified 
in Section 3. 

1.2 Notations 



Notation 


Comments 


PId 


the set of physical process identifiers (e.g., IP address and port number) 


Xld 


the set of logical identifiers; <ik < N, for any ik 6 Xld 


GId 


the set of general identifiers of the form gk = (ik, Xk), where ik 6 PId,Xk € Xld 


XtoP, PtoX 


one-to-one correspondence from Xld to PId, and its inverse. 


succ(x, k, R) 


the k th successor of x in ring R 


pred(x,k, R) 


the k th predecessor of x in R 


succset(x, k, R) 


successor set of x; succset(x, k, R) = {succ(x,t, R) : < £ < k} 


psuccset(x, k, R) 


proper successor set of x; psuccset (x, k, R) = {succ(x,£,R) : 1 < £ < k} 


pre,dset(x, k, R) 


predecessor set of x; predset(x, k, R) = {pred(x,£, R) : < £ < k} 


ppredset(x, k, R) 


proper predecessor set of a:; predset(x, k, R) = {pred(x, £, R) : 1 < £ < k} 


block(x, k, R) 


block of a:; block(x,k, R) = succset(x, k, R) U predset(x,k, R) 



Table 1 : Notations used in this paper. 

Table 1 shows the main notations used in this paper. Each process is identified by a physical identifier (e.g., IP 
address and port number), and a logical identifier in identifier space 0..2™ — 1, where N = 2™. A ring R is a nonempty 
subset of logical identifiers (Xld), ordered in a clockwise direction. 

The k th successor of x in R is denoted by succ(x, k,R). For k = 0, succ(x,0, R) = x if x £ R, and is otherwise 
undefined. If A; > 1 then succ(x, k, R) is the k th value encountered when moving clockwise in R — x starting from the 
position of a;, if \R — x\ > k, and is otherwise undefined. The k th predecessor of a; is defined similarly (see Table 1). 

2 The MultiChord Protocol 

In this section we present the details of the MultiChord protocol. 



2.1 Process Automaton: Signature 

For the rest of this section, we fix a physical address i € PId, and describe the process automaton for location i, 
MultiChordi. Throughout this section, we use me as an abbreviation for the general identifier GId g with g.phys = i 
and g.log = PtoX(i), where g.phys and g.log denote the physical identifier, and the logical identifier of g, respec- 
tively. Formally, MultiChordi 1S a timed I/O automaton, as defined in Chapter 23 of [3]. 

The signature of MultiChordi is given in Figure 1 . The external signature describes the inputs and outputs (primar- 
ily, client invocations and responses) by which the MultiChord service interacts with its environment. The external 
signature includes join, lookup and receive inputs and corresponding acknowledgments. We do not include special 
"leave" requests and responses in this paper; instead, we treat leaves as failures. We do not consider rejoining after a 



failure. The internal signature consists of transitions that implement join and lookup protocols, and maintain the finger 
tables in the face of a limited rate of change. 



Input: 
join( J) i, J a finite subset of PId — {i} 
lookup(a;)j, x £ Xld 
receive(m)j j j, m £ Msg, j £ PId 

Output: 
join-ack^ 

lookup-ack(ff) { ,H C GId 
send(m)jj,m £ Msg, j £ PId 



Internal: 
join-pingj 
neigh bor-refresh^ 
chord-ping^ 
stabilize(a;)j, x £ Xld 
garbage-collect(/)j, / £ Finger 

Time-passage: 

time-passage(t), t £ R + 



Figure 1: MultiChordi : Signature 

2.2 Process Automaton: Data Types and Constants 

Table 2 shows the data structures and the message formats used by the MultiChord protocol. In addition we define two 
operations on sets of fingers: 

1. update(F, F'), which computes F U F'\ if a finger / belongs to both sets of fingers F and F' then / inherits 
the highest expiration time, exptime, that it has in the two sets. 

2. truncate(F, t), which bounds the exptime of each finger f £ F tot, i.e., / .exptime : = max(/ .exptime, t). 



Notation 


Comments 


Finger 


finger data structure; consists of fields: (gid 6 GId, exptime 6 R- u U {oo}) 


Reqld 


request identifier set, partitioned into subsets Reqld(i), i 6 PId; used to identify lookup instances 


Request 


used to implement one lookup stage; consists of fields: (id £ Reqld, stage £ N + , target £ Xld) 


JoinRecord 


used to keep track of progress in a process' attempt to join the system; consists of 
fields: (reqids C Reqld, comp C Reqld, acktime 6 R-° U {oo}) 


ClientRecord 


used to keep track of client-initiated lookup requests at a particular location; consists of fields: 
(reqids C Reqld, comp C Reqld, acked C Reqld(i)) 


LookupMsg 


lookup message; consists of fields: (tag = lookup, id 6 Reqld, stage £ N + , target £ Xld) 


LookupRespMsg 


lookup response message, (tag = lookup-resp, id 6 Reqld, stage £ N + ,preds £ Set[Finger]) 


Lookup CompMsg 


lookup completion msg., (tag = lookup-completion, id 6 Reqld, stage £ N + , block 6 Set[Finger]) 


PingMsg 


ping message used to refresh finger information, (tag = ping, ping) 


BlockMsg 


message used to send a block to another message, (tag = block, block 6 Set[Finger]) 


T e 


the timeout value for expiration of entries in the finger table 


T g 


the time between scheduling gossiping messages, i.e., PingMsg and BlockMsg messages 


Tj 


the time from when a joining process has received all its responses until it responds to its client 


b 


number of proper predecessors and successors that a process maintains about itself 
and its power-of-two fingers 


c 


number of responses that a client returns in response to a lookup request; c < b 



Table 2: Data structures and message formats used in MultiChord. 

MultiChord uses only five types of messages: Lookup, LookupRespMsg and LookupResp Completion to imple- 
ment join and lookup operations, and PingMsg and BlockMsg to maintain the finger tables in the face of changes. 

In addition, MultiChord uses the following time constants: (1) T g , the time between scheduled gossiping messages, 
(2) T e , the timeout value for expiration of entries in the finger table, and (3) Tj, the time from when a joining process 
has completed its systematic collection of responses until it responds to its client. 

Finally, MultiChord uses two constants b and c. Constant b represents the level of redundancy used by a process to 
maintain routing information. In particular, each process maintains its b proper successors and b proper predecessors, 
and b proper successors and b proper predecessors of each of its power-of-two fingers. Constant c represents the level 
of redundancy used to perform lookups, the basic operations in MultiChord. During lookup operations, each process 
issues c concurrent queries, which makes it highly likely that at least one process will respond. The value of c is chosen 



to be larger than the number of changes that are likely to occur in an arc of the ring, in intervals of some reasonable 
length. The length of this interval should be sufficiently long to allow recovery from recent changes. The value of b is 
usually larger than c; c must be large enough to ensure a response under "normal" conditions (with bounded changes), 
while b must be large enough to support the infrastructure maintenance protocol. 



2.3 Process Automaton: State 

The state of MultiChord , consists of the state variables listed in Figure 2. Note that our initializations of these 
variables assign tuples to record-valued variables. We use the convention that the order of the components in the 
tuples is the same as the order presented in the definitions of the record types. 



State variables: 

status G {idle, joining, active}, initially idle 
join G JoinRecord , initially (0,0, oo) 
client G C'lientRecord, initially (0, 0, 0) 
used-reqids C Reqld(i), initially 
requests G Set[Request], initially 
fingers G Set[Finger], initially {(me,oo)} 
out-queue, a sequence of Msg X PId, initially empty 

ping-time G R-° U oo, initially oo 
nbr-refresh-time G R-° U oo, initially oo 
failed, a Boolean, initially false 



Derived variables: 

local-ring = {x G Xld : 3/ G fingers[f.log = x]} 

For x G Xld, k > 0: 
f-succset(x,k) = {/ G fingers : f.log G succset(x,k, local-ring)} 
f-psuccset(x,k) = {/ G fingers : f.log G psuccset(x, k, local-ring)} 
f-predset(x,k) = {/ G fingers : f.log G predset(x,k, local-ring)} 
f-ppredset(x,k) = {/ G fingers : f.log G ppredset(x,k, local-ring)} 
f-block(x,k) = {/ G fingers : f.log G block(x,k, local-ring)} 



Figure 2: MultiChordi : State 

The status variable keeps track of the state of process i. The join variable keeps track of the progress of the joining 
protocol for process i, and the client variable keeps track of the progress of all client-initiated lookups at location i. 
The fingers variable contains a set of fingers, which represent process i's best knowledge of the current members 
of the ring (including their expiration times). The used-reqids variable keeps track of which request identifiers in 
Reqld(i) have already been used; it is used to model the generation of unique identifiers. The requests variable keeps 
track of the set of requests that have been initiated at location i; these may be generated on behalf of the local joining 
protocol, local client lookup requests, or heavyweight stabilization. The out-queue variable is a buffer for messages 
that process i has generated and has not yet sent. 

The nbr-refresh-time and ping-time variables are used to schedule the gossip messages; nbr-refresh-time is 
used by process i to schedule sending of its own block to its nearby neighbors, whereas ping-time is used by process 
i to schedule "ping" messages to request block information from other processes. Finally, the failed variable is a flag 
saying whether process i has failed. 

Process i also maintains some derived variables, which also appear in Figure 2. The derived variable local-ring is 
defined to be the set of logical identifiers that appear in i's fingers variable, that is, local-ring represents i's current 
local view of the global ring. Other derived variables are defined to give various successor and predecessor sets, with 
respect to the local-ring. For example, f-succset(x, k) is defined to be the set of fingers in the current finger set whose 
logical identifiers are among the k successors of x in the current local-ring; if x appears in local-ring then this set 
include x itself. 

2.4 Process Automaton: Transitions 

In this section we present the main transitions in MultiChord. Section 2.4.1 describes the basic transitions such as 
message sending, garbage-collection, and time-passage transitions. Section 2.4.2 shows the transitions involved in the 
joining protocol, and Section 2.4.3 presents the transitions involved in the stabilization protocol. Finally, Section 2.4.4 
describes transitions involved in the client lookup protocol. 



2.4.1 Basic Transitions 

Figures 3(a)-(c) shows three basic transitions: sending, garbage-collection, and time-passage transitions. 

A send transition simply removes the first Msg from out-queue and sends it to the indicated destination, using an 
assumed point-to-point network. Process i can do this only if it has at least begun the protocol, and has not failed. 
A garbage-collect transition removes an entry from its fingers set when the entry's exptime has been reached. A 



time-passage transitions advances the time until the next event, i.e., scheduling times of pinging, acknowledging 
the client, or neighbor-refreshing, and the exptime of any finger in the fingers set. Time may not pass at all if the 
out-queue is nonempty; this implies that messages in the out-queue are sent out immediately, without any time 
passage. 

Output send(m) (M) Internal garbage-collect(/); time-passage(t) 

Precondition: Precondition: ' Precondition: 
--failed -.failed if ^failed then 

status ± idle status = active now +t < ping-time 

(m,j) = head{out- queue) f e fingers now +t < join. ack-time 

Fffpp,. t „ -,„•„,„ s „„„, now +t < nbr-refresh-time 

EJieci. f .exptime < now — ' 

remove head(out-queue) Effect- V -f e fingers :now+t< f. exptime 

fingers := fingers - {/} out-queue is empty 

Ettect: 
( a ) (b) now:=now+t,. 

Figure 3: (a) Sending transitions; (b) Garbage-collection transition; (c) Time passage transitions. 



2.4.2 Transitions Involved in the Joining Protocol 

Like Chord, in MultiChord a process uses lookups to populate its finger table when it joins the system. Where the two 
protocols differ is in the amount of state required to join the system. Whereas in Chord a process is required to know 
only a set of successor processes, in MultiChord a process is required to know a set of processes (i.e., a 6-block) for 
each of its power-of-two fingers. As we will show in Section 3 this redundancy increases the resilience of the protocol 
in the face of changes. 

Next, we present the details of the transitions involved when process i joins the system. These include: 

1. The join^ transitions, by which the client at location i requests to join. 

2. The receive transitions for lookup, lookup-resp, and lookup-comp messages, which are involved in initially 
populating process i's finger set. 

3. The join-ping transitions and the receive transitions for ping messages; these are used to complete the finger set 
before process i responds to the client. 

4. The join-ack^ transitions, by which process i responds to its client. 

Figure 4 shows the join and join-ack transitions. In a join ( J) , transition, processor i initiates joining by submitting 
a set J of PIds of other processes that should already be members of the system. Process i handles the join request 
only if it has not failed and has not previously begun joining. To handle the join request, the process first sets its status 
to joining and schedules its ping task. If J = 0, the process is already done and schedules its response to the client. 
Otherwise, if J ^ 0, process i launches a set of lookup requests, one for itself and one for each of its power-of-two 
successors. 

When all these requests have completed, and when sufficient additional time has passed (as determined by a 
scheduled ack-time being reached), process i can report back to the client with a join-ack^ transition. When it does 
so, it converts its status to active and schedules its nbr-refresh task. 

As in Chord, MultiChord implements an iterative lookup protocol. The processing of a lookup request involves 
three types of transitions, which appear in Figure 5. When process i receives a lookup message, it handles this message 
only if it is already active, that is, if it has completed its joining protocol. In order to handle the lookup message, it 
sends either a lookup-resp or a lookup-comp message, depending on whether it thinks that the search has reached its 
goal. The test for completion is that, according to i's current information, target x is among the c proper predecessors 
of the target. In the case of a lookup-comp message, process i sends back its best information about the target's block 
of radius b. In the case of a lookup-resp message, process i sends back its c best proper predecessors for the target. ' 

When process i receives a lookup-resp message for the current stage of a current request, it updates its finger 
table with the information contained in the preds field of the incoming message. Then because the request is not 
completed, process i generates a new batch of lookup messages for the next stage of the same request. This next stage 



1 In either case, process i first truncates all fingers' exptimes to now plus the maximum timeout value T e ; this is because i's entry for itself has 
exptime = oo, but we do not want others to record exptime = oo for i. 



Input join (J), 
Effect: 

if ^failed then 

if status = idle then 
status := joining 
ping-time := now 
if J = then join . acktime := now 
else 
for a; £ {me. log} U {me. log + 2* : < k < n — 1} do 
choose rid £ Reqld(i) — used-reqids 
used-reqids := used-reqids — {rid} 
join.reqids := join.reqids U {rid} 
requests := requests U {(rid,l,x)} 
for j £ J do 

add ((lookup, rid, l,!),.?) to out-queue 



Output join-ackj 
Precondition: 

^failed 

status = joining 

join.reqids C join.comp 

join. acktime = now 
Effect: 

status := active 

nbr-refresh-time := now 



Figure 4: Client-level transitions related to joining 



Input receive(lookup, r, s,x)jj 
Effect: 

if ^failed then 

if status = active then 

if me. log £ ppredset(x, c, local-ring) then 

block := truncate(f-block(me.log , b), now + T e ) 
add ((lookup-comp, r, s, block), j) to out-queue 
else 
preds := truncate(f-ppredset(x , c), now + T e ) 
add ((lookup-resp,r, s,preds), j) to out-queue 

Input receive(lookup-resp, r,s,F)j^ 
Effect: 

if ^failed then 

new-fingers := {f £ F : f.exptime > now}) 
fingers := update( fingers, new-fingers) 
if3x[(r,s,x) £ requests)] then 
choose x where (r, s, x) G requests 
requests := requests — {(r, s,x)} U {(r, s + 1,3;)} 
f° r / £ f-ppredset(x , c) do 

add ((lookup, r, s + 1, i), f.gid.phys) to out-queue 



Input receive(lookup-comp, r, F)j^ 
Effect: 

if ^failed then 

new-fingers := {/ £ F : f.exptime > now}) 
fingers := update(fingers , new- fingers) 
if r £ join . reqids then 
join.comp := join.comp U {r} 

if join .reqids C join.comp and join, acktime = oo then 
join. acktime := now + Tj 
if r £ client . reqids then 

client. comp := client. comp U {r} 



Figure 5: Transitions of the lookup protocol 



has the next-higher stage number, which is recorded in the request record. The messages for the new stage are sent 
to the c currently-known best proper predecessors of the target. Note that the number of messages does not increase 
exponentially at each stage; the protocol limits the the number of messages to c. 

When process i receives a lookup-comp message for the current stage of a current request, it updates its finger table 
with the information in the block field of the incoming message. As in the lookup-resp case, process i increments the 
request's stage number, to register the fact that some response for this stage has arrived. If the current request is part of 
i's joining protocol, then the completion of this request is recorded in the join record; if this represents the completion 
of the last request, then process i also schedules the client acknowledgment. On the other hand, if the request is being 
done on behalf of a client-initiated lookup, the completion is recorded in the client record (see Appendix A). 

During the joining protocol, process i periodically pings its power-of-two fingers for their 6-blocks. The relevant 
transitions are the join-ping transitions and the receive transitions for ping messages and their responses (see Figure 6). 

Process i performs a join-ping transition while it is joining, whenever ping-time is reached. When it does so, 
it sends ping messages to the c-blocks of all targets for which lookup requests have already completed. This allows 
process i to augment and refresh its information about completed requests while finishing the joining protocol. When 
process i receives a ping message, it responds by sending back its 6-block, in a block message. When process i receives 
a block message, it updates its finger table with the new information. 



Internal join-ping^ 
Precondition: 

^failed 

status = joining 

ping-time = now 
Effect: 

for r G requests where r.id G join.comp do 
for/ 6 f-block(r. target, c) do 

add ((ping), f.gid.phys) to out-queue 

ping-time := now + T 9 



Input receive(ping) Jj j 
Effect: 

if ^failed then 

if status = active then 

6/ocA := truncate(f-block(me.log,b),now + T e ) 
add ((block, block),]) to out-queue 

Input receive(block, -F)^ 
Effect: 

if -^failed then 

if status = active then 

new-fingers := {/ G -P : f.exptime > now} 
fingers := update( fingers, new-fingers) 



Figure 6: Transitions related to pinging during the join protocol 



2.4.3 Transitions Involved in Stabilization 

Once process i is active, it performs several types of transitions to maintain its finger table. The protocol includes two 
kinds of stabilization: normal case, lightweight stabilization, and a heavier-weight stabilization. 

In the lightweight stabilization protocol, process i periodically sends its 6-block to its nearby neighbors (the mem- 
bers of its 6-block), and periodically pings processes in the vicinity of its power-of-two successors, so that they send 
i their current 6-blocks. The transitions involved in this lightweight stabilization protocol are the neighbor-refresh 
transitions, the chord-ping transitions, and the receive transitions for ping and block messages. Note that the pseu- 
docode for ping and block transitions has already been presented in Figure 6, while the pseudocode neighbor-refresh 
and chord-ping transitions appears in Figure 7(a)-(b). 



Internal neighbor-refresh,, 
Precondition: 

^failed 

status = active 

nbr-refresh-time = now 
Effect: 

f° r / £ f-block(me.log, b) do 
add ((block, f-block(me .log, &)), 
f.gid.phys) to out-queue 

nbr-refresh-time := now + T g 

(a) 



Internal chord-ping^ 
Precondition: 
^failed 

status = active 
ping-time = now 
Effect: 

for k G {0, ...,n- 1} do 

f° r / £ f-block(me.log + 2 k , c) do 
add ((ping), f.gid.phys) to out-queue 



ping-time : 



(b) 



Internal stabilize(a;)j 

Precondition: 
^failed 
status = active 

Effect: 

choose rid G Reqld(i) — used-reqids 
used-reqids := used-reqids — {rid} 
requests := requests U {(rid, l,x)} 
f° r / £ f-ppredset(x, c) do 

add ((lookup, rid, 1, x), f.gid.phys) 
to out-queuE , 



Figure 7: Transition related to stabilization. 

In the heavyweight stabilization protocol is similar to the Chord stabilization protocol (see Figure 7(c)). Process i 
(for any reason, unspecified here) may try to obtain new information about any target x. Most commonly, such a target 
will be one of its power-of-two successors. For example, process i might execute stabilize^), for each x of the form 
PtoX(i) + 2 k , at regular intervals, or when it suspects that its information is out-of-date. 



2.4.4 Transitions Related to Client Lookups 

The transitions related to client-initiated lookup operations include the receive transitions already described, plus the 
lookup and lookup-ack transitions. These last two appear in Figure 8. 

When process i receives a client-initiated lookup request, it handles it in much the same way it handles a request 
in the joining protocol. Namely, it chooses and records a request identifier, and sends a lookup message to each of 
the c best proper predecessors it knows for the target identifier. An exception: If process i believes it is one of the c 
best predecessors, it does not bother sending out any lookup messages, but simply records the fact that the lookup is 
done. A lookup-ack can occur when a request is done but not yet acknowledged to the client. In this case, the response 
includes information about process i's current c best predecessors for the target. 



Input lookup(a;)j 
Effect: 

if ^failed then 

if status = active then 

choose rid G Reqld{i) — used-reqids 

used-reqids := used-reqids — {rid} 

if me. log G ppredset(x, c, local-ring) then 

client. comp := client. comp U {rid} 
else 

client. reqids := client. reqids U {rid} 
requests := requests U {(rid,l,x)} 
f° r / £ f-ppredset(x, c) do 

add ((lookup, rid, l,x), f.gid.phys) to out-queue 



Output lookup-ack(i?)j 
Precondition: 

^failed 

r G requests 

r.id G client. comp — client. acked 

H = {f.gid : f G f-ppredset(r. target, c)} 
Effect: 

client, acked := client. acked U {r.id} 



Figure 8: Transitions for client lookup 

3 Summary of Analysis Results 

In this section we give a short and informal summary of our analysis results. Appendix A presents the proofs of these 
results. 

We make the following assumptions about the environment: (1) all processes are time-synchronized, (2) the mes- 
sage delay is bounded above by d, and there is no message loss, (3) during an interval of time T g + 2d, the number of 
join-ack events among processes in an "arc" of the ring containing at most 6 + 1 processes is at most joinbd, and (4) 
during an interval of time T e , the number of failed processes in an "arc" of the ring containing at most b+1 processes 
is at most failbd. 
Then we show that if these assumptions hold, and furthermore, if the following constraints are satisfied: 

1. Tj>T g + 2d and T e > 5(T g + 2d) 

2. c > 7 joinbd + Afailbd 

3. b >2c + 3joinbd + m&x(2joinbd, failbd) 

we prove that all lookup operations are correct. In particular we prove the following result: 

Theorem 3.1. Every good execution a satisfies 2T g + Qd-lookup-correctness. 

The notion of e-Lookup-correctness is defined as follows: suppose that a lookup-ack(i : f)j event occurs in /? at time 
t, in response to a prior lookup(a;)j. Let /3 1 be the portion of /3 ending with the given lookup-ack(i : f)j event. Then 
there exists a ring R such that: 

1. R C aug-ring(P'), 

2. global-ring (P 1 ) — {PToX(j) : join-ack occurs at a time > t — e} C R, and 

3. H = ppredset(x, c, R). 

Furthermore, we show that in the absence of any other events in the system the lookup latency is bounded. More 
formally, we prove the following result: 

Theorem 3.2. Suppose that a is a good execution, a 1 a finite prefix of a containing at least 2c + 1 join-ack events. 
Suppose that: 

1. The final step of a 1 is a lookup, step in which i initiates request r, with target x. 

2. No other requests (on behalf of joins, client lookups, or stabilizes) are active at any time > £time(a') — T e . 
Then request r terminates with a receive(lookup-comp) step, at a time that is < £time(a') + 4(log N + \)d. 

In order to prove these results, in Appendix A we first prove a series of results asserting that the basic routing 
infrastructure is maintained correctly by the joining and refresh protocols. 

While the deterministic assumptions on the bounded number of joins {joinbd) and failures {failbd) allow us to 
prove strong analytical results, these assumptions are not always realistic. We consider this issue in Appendix B, 
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Figure 9: (a) The lookup failure versus the rate of change; (b) the average path length and the 90-th percent confidence 
interval as a function of change rate. 

where we give bounds on the probability that these assumptions hold in a steady state system in which processes join 
according to a Poisson process and have a lifetime drawn from an exponential distribution. In particular, we compute 
the mean time between two violations of these assumptions as 



Tf> 



T 



3 

N 



(c/3-\Tj(b+l))-> 
4AT 3 .(6+1) 



(1) 



where A represents the normalized rate of change (i.e., the rate of change in the entire system divided by the number 
of processes N in the system), c/3 > \Tj(b + 1), and b > 13c/6. 

4 Simulation Results 



In this section we evaluate our algorithm by simulation. Our goal is twofold. First, we want to get a sense of how 
much we can push the protocol in practice before it breaks, i.e., before we start to see lookup failures. Second, we 
want to see how the protocol performs on the average case. We use the average number of stages in a lookup as the 
metric to evaluate the performance of MultiChord. 

We have developed an event driven simulator that accurately implements the protocol at the message level. In all 
simulations, we use T g = 10 sec, Tj = 11 sec, and T e = 55 sec. The message propagation delay is bounded by d = 50 
ms. Note that these values satisfy the constraints presented in Section 3, i.e., Tj > T g + 2d and T e > 5(T g + 2d). 
Each process schedules heavy stabilization every 60 sec. 

We consider a network with 1, 000 processes, in which processes join at a rate A a according to a Poisson process, 
and have an exponentially distributed lifetime with the mean N/X a ; thus, the number of processes in the system 
remains roughly the same. In addition, we assume that the system receives lookups at a rate approximately 10 times 
larger than the join and failure rate, r. 

Figure 9(a) plots the lookup failure rate versus the arrival rate of new processes in the system (i.e., rate of join) 
over 10, 000 lookups. During each simulation there are approximately 1000 new processes that join the system, and 
1000 processes that fail. We consider two cases: (i) c = 2, b = 5, and (ii) c = 4, b = 9. As expected, the rate of 
lookup failure increases as the join rate increases. However, increasing the level of redundancy (i.e., parameters b and 
c) makes a significant difference. While in case (i) we did not record any lookup failure for join rates less or equal to 
0.1, in case (ii) we did not see any lookup failure for a join rate five time larger, i.e., 0.5. Furthermore, for a join rate 
of 2.0 the rate of lookup failure in the first case is about 18 times larger than in the second case. 

It is interesting to compare the simulation results with our upper bound on the mean time Tf between two violations 
of the deterministic constraints. Consider the first case where c = 2 and 6 = 5. Using Eq. (1), for a join rate of 0.5 
we obtain Tf = 14 ms. 2 This is a very small value given the fact that a lookup operation is generated every 50 ms 
(i.e., there are roughly 10 lookups for every join operation). One explanation for this large discrepancy is that a single 
constraint violation will hurt only a small fraction of lookups, if at all. Indeed, the lookups that do not use the region 
of network where the constraints are violated will not be affected. 



Here we use c : 



5, A = 1/1000 (there is one join and one failure every 0.5 sec on average and N = 1000), and Tj = 11 sec. 



Figure 9(b) plots the average number of stages (path length) of a lookup versus the rate of join for (i) c = 2, b = 5, 
and (ii) c = 4, b = 9, respectively. There are two points worth noting. First, the average path length is significantly 
smaller than in Chord; in Chord, the expected path length is log N/2, which in our case translates to 5 hops. This is 
because in MultiChord every process maintains a much larger set of fingers than in Chord. This increases the chance 
that a MultiChord process will know fingers closer to the target than an equivalent Chord process, which ultimately 
will reduce the number of lookup stages. Second, as the join rate increases, the lookup path length decreases slightly. 
To understand this recall that in steady state the average life time of a node is N/X a where A a is the join rate. However, 
it takes a process at least T g time to join the system. Thus a node will be inactive for at least / = T g X a /N of its life 
time, which means that at least fN processes in the system would be inactive on an average. As the join rate increases, 
the fraction / of inactive nodes increases, which will lead to a corresponding reduction in the number of active nodes 
in the system. A secondary reason is that as the join rate increases so does the failure rate. Since we do not report 
failed lookups, and since the failed lookups tend to have more stages, the reported path length is an underestimation. 

5 Conclusions and Future Work 

In this paper we present MultiChord, a namespace management algorithm based on Chord [7]. MultiChord uses 
redundancy and lightweight mechanisms to accommodate limited changes in time and space. We analyze MultiChord 
and show that lookups are guaranteed to be successful and furthermore that the lookup latency is bounded. 

It would be interesting to analyze the behavior of the algorithm in situations that are less well-behaved than what 
we have described in this paper. In particular, we plan to consider what happens if the rate of change exceeds our 
assumed bound for some part of the execution, but at some point "stabilizes" to obey the rate bound. In such cases, 
we believe that our algorithm will eventually stabilize to a nearly-ideal state. It remains to determine if this is so and 
determine bounds on how long this might take. 
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6 Appendix A: Analysis 

In this appendix we prove the results which were summarized in Section 3. 

Let /3 by a finite sequence of external actions of MultiChord, according to the external signature just defined. Then 
we define the global ring after (3, global-ring (/?), to be the set of Xlds x such that a }o\n-ack xtoP i x \ event occurs in /3 
and no fa'i\xtoP(x) occurs in /3. That is, the global ring after /3 consists of those processes that have completed joining 
the system and have not failed. We extend this same definition to finite executions of untimed or timed automata that 
have the given external signature. 

If /3 is a finite timed sequence of actions in the MultiChord external signature, then we define the augmented ring 
after /3, aug-ring(/3), to be global-ring (/?) U X, where X is the set of Xlds x such that faWxToP(x) occurs in /3 at a 
time > ltime(/3) — T e . That is, aug-ring(a) augments global-ring (/?) by adding in the logical identifiers of recently 
failed processes. Again, we extend this definition to finite executions of timed automata that have the given external 
signature. 

6.1 Service Guarantees 

We describe safety and latency guarantees. We do not present any liveness guarantees here, replacing them with 
latency guarantees. 

6.1.1 Safety 

The following condition is simple a well-formedness condition, expressing basic conditions such as "the service re- 
sponds only to invocations that were actually made". 

• Well-formedness: For each i, at most one join-ack^ occurs in /3. Any join-ack^ in /3 is preceded by a join(*)j. 
Any lookup-ackj is preceded by a lookup(*), with no intervening lookup-ack(*)j. If fail, occurs in /3, then no 
following outputs occur. 

We have not formulated any interesting safety guarantees related to joining. For client lookup, we require the 
following property, parameterized by e € R-°: 

• e-Lookup-correctness: Suppose that a lookup-ack(i : f)j event occurs in (] at time t, in response to a prior 
lookup(a;)j. Let /3 1 be the portion of ending with the given lookup-ack(i : f)j event. Then there exists a 
ring R such that: 

1. R C aug-ring(P'), 

2. global-ring (/3 1 ) — {PToX(j) : join-ack occurs at a time > t — e} C R, and 

3. H = ppredset(x, c, R). 

6.1.2 Latency 

As noted above, we replace liveness claims by latency bounds: 

• e- Join-latency: Suppose that a join( J)j event occurs in /3, at time t. 

1. If J = then a corresponding join-ack^ occurs at time t. 

2. If there exists j 6 J such that join-ack occurs before the join( J)j, and neither i nor j fails in /3, then a 
corresponding join-ack^ occurs by time t + e. 

• e-Lookup-latency: If a lookup(a;)j event occurs in /3 at time t and no fail, occurs in /3, then a corresponding 
lookup-ack(*),; occurs by time t + e. 

6.2 Assumptions for Analysis 

In this section we formalize the algorithm constraints and the assumptions about the environment, which we discussed 
in Section 3 
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6.2.1 Restrictions on the algorithm 

Constraints on values of the constants 6, c, d, T g ,T e , and Tf. 

• T J >T g + 2d 

. T e > 5(T g + 2d) 
Scheduling assumptions: 

• The locally controlled actions that are enabled are performed without any intervening time-passage. 

6.2.2 Restrictions on the environment 

Constants: 

For the purpose of analysis, we introduce two constants, joinbd and failbd. We assume: 

• c > 7 joinbd + 4failbd 

• b > 2c + hjoinbd 

• b > 2c + 3joinbd + failbd 
Restrictions on timing and failures: 

• No message loss. 

• No time passes while a locally-controlled action is enabled. 

• Bounded local joins: An execution a satisfies bounded local joins provided that for any finite prefix a 1 of a, the 
following holds. 

Let x,y £ Xld where \global-ring(a') (~l [x,y]\ < b + 1. Then the number of join-ackj. events that occur in a 1 
at times > Itime(a') - (T g + 2d), where PToX(k) € [x, y], is < joinbd. 

That is, at any point in the execution a, the number of recent join-ack events among processes in an "arc" of the 
ring containing at most 6 + 1 processes is at most joinbd. 

This assumption is not ideal because it is expressed in terms of the number of join-ack events, which are under 
the control of the algorithm (rather than the environment). We could justify this assumption in terms of a more 
primitive assumption that bounds the rate of join events, which are controlled by the environment. To do this, 
we might need to modify the algorithm so that it schedules the join-acks so that (in the normal case) they occur 
a fixed amount of time after the joins. Alternatively, a probabilistic justification might be possible. 

• Bounded local failures: An execution a satisfies bounded local failures provided that for any finite prefix a 1 of 
a, the following holds. 

Let x, y € Xld where \global-ring(a') f][x,y]\ < b + 1. Then the number of fail^ events that occur in a' at 
times > Uime(a') - T e , where PToX(k) G [x,y], is < failbd. 

That is, at any point in a, the number of recent fail events among processes in an arc of the ring containing at 
most 6 + 1 processes is at most failbd. 

We also need a special assumption to ensure that there are "enough" processes in the ring. 

• Enough-processes An execution a satisfies enough-processes provided that it has a finite prefix a 1 such that: 

1. At least 26+1 join-ack events occur in a'. 

2. No fail event occurs in a 1 . 

3. In any state of a after a 1 , the total number of live processes is always > 26 + 1. 

We call the shortest such prefix a 1 the initialization prefix. 
A good execution is one that observes all the timing and failure restrictions given in this section. 
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6.3 Basic Lemmas 

The first lemma says that exptimes of fingers are always > now. 

Lemma 6.1. The following is true in any state that is reachable in a good execution: 
If f G fingers i then f.exptime > now. 

The next lemma says that every physical identifier i that appears in another process' fingers set, or in a message 
in transit, must correspond to a process whose status is active. 

Lemma 6.2. The following is true in any state that is reachable in a good execution: 
Suppose that f € Finger, f.phys = i, and any of the following holds: 

1. f G fingers j for some j ^ i. 

2. f (E m. block for some m <E BlockMsg that is in transit. 

3. f (E m.preds for some m € LookupResp in transit. 

4. f G m. block for some m € LookupComp in transit. 

Then statusi = active. 

The next lemma says that, if a process fails at a time t, then no expiration time for that process that is greater than 
t + T e ever appears anywhere in the state. 

Lemma 6.3. Suppose that a is a finite execution, and fail, occurs at time t in a. Suppose that f (E Finger and 
f.phys = i. Suppose that, in £state(a), any of the following holds: 

1. f G fingers, for some j ^ i. 

2. f (E m. block for some m <E BlockMsg that is in transit. 

3. f (E m.preds for some m (E LookupResp in transit. 

4. f <E m. block for some m (E LookupComp in transit. 

Then f.exptime < t + T e . 

As a corollary to some of the previous lemmas, the following lemma says that a process that has failed more than 
T e time ago does not appear in anyone's fingers set. 

Lemma 6.4. Suppose that a is a finite execution, and fail, occurs strictly before time £time(a) — T e in a. Suppose 
that f € Finger and f.phys = i. Then in £state(a), f does not appear in fingers j for any j ^ i. 

Proof. By contradiction. Suppose that in lstate(a), f € fingers, for a particular j ^ i. Then by Lemma 6.3, in 
lstate(a), f.exptime < t + T e , where t is the time at which fail, occurs. Lemma 6.1 implies that, in lstate(a), 
f.exptime > now, that is, f.exptime > £time(a). These two inequalities together imply that ttime{a) < t + T e . 
This contradicts the hypothesis that fail,: occurs strictly before time ltime(a) — T e . □ 

6.4 Maintaining Neighbor Sets 

In this section, we prove that the neighbor sets are properly maintained. We divide the work into three steps: First, 
we consider what happens when there are no failures and only a bounded number of joins. Second, we consider the 
general case, with unlimited failures and joins. 

The results we prove express knowledge guarantees for live processes. Specifically, we show that all live processes 
always know about all neighbors that joined more than time 2T g + 5d ago. Moreover, after a process has been live for 
sufficiently long, it knows about all neighbors that joined more than time d ago. 

Breaking the proof up in some such way seems necessary in order to make the proof tractable. Each stage intro- 
duces its own new difficulties: the first stage already includes many of the issues involving the timing of the flow of 
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information during and soon after the joining protocol. The second stage introduces issues of local knowledge — each 
process maintains information about its local neighborhood only. The third stage introduces the complications of 
failures, which mean that a process cannot rely on responses from any particular other process. 

We expect this decomposition to be useful in constructing the general proof, because the ideas of the first stages 
should be useful in the later stages. Also, the result for the first stage should be directly usable in proving the more 
general results, in describing properties of the initial set-up phase. 

6.4.1 Basic lemmas 

The following lemma says that every block message contains a high expiration time for the sender. 

The mention of a deadline for a message in transit refers to a detailed state-machine model for a timed channel, in 
which a deadline is explicitly kept for each message. This deadline is described in terms of absolute time. 

Lemma 6.5. Let a be a good finite execution. If a block message is in transit from i with deadline £ then it contains a 
finger for i with exptime > £ + T e — d. 

6.4.2 No failures, limited joins 

In the case we consider in this subsection, no processes fail and at most 2c + 1 join-acks occur. With this limited 
number of join-acks, every process is in every other process' c-block, so we do not have to worry about issues of local 
knowledge. 

The following lemma says that everyone "always" has a finger for io, with a "sufficiently high" expiration time. 
The precise statement of this is rather complicated, because many different cases are covered. 

Lemma 6.6. Let a be a good finite execution that contains no fail events, and contains at least one and at most 2c + 1 
join-ack events. Let io denote the process that performs the first join-ack in a. Let i € PId. 
Then in £state(a): 

1. If status i = joining and a contains a receive(lookup-comp)*j event for target PToX(i), then there exists 
f G fingers i with f.phys = io such that: 

(a) One of the following holds: 

i. f .exptime > ping-time i + 2d. 

ii. There is a ping message in out-queue i addressed to io and f. exptime > now + 2d 
Hi. There is a ping message in transit from i to io with deadline £ and f .exptime > £ + d. 
iv. There is a block message in out-queue^ addressed to i, and f. exptime > now + d. 

v. There is a block message in transit from io to i with deadline £, and f .exptime > £. 

(b) f. exptime > now. 

2. If status i = joining and a contains a receive(block)j ,,; event, then there exists f € fingers i with f.phys = io 
such that: 

(a) One of the following holds: 

i. f .exptime > ping-time i + 2T g + Id. 

ii. There is a ping message in out-queue i addressed to io and f. exptime > now + 2T g + 7d. 
Hi. There is a ping message in transit from i to io with deadline £ and f .exptime > £ + 2T g + Qd. 
iv. There is a block message in out-queue i addressed to i, and f .exptime > now + 2T g + 6d. 

v. There is a block message in transit from io to i with deadline £, and f .exptime > £ + 2T g + 5d. 

(b) f .exptime > now + 2T g + 5d. 

3. If status i = live then there exists f € fingers i with f.phys = io such that: 

(a) One of the following holds: 
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i. f.exptime > nbr-refresh-time i + 2T g + 5d 

ii. There is a block message in out-queue i addressed to i$, and f.exptime > now + 2T g + 5d 
Hi. There is a block message in transit from i to io with deadline £ and f. exptime > £ + 2T g + Ad. 

iv. There is a finger for i in fingers t with exptime > nbr-refresh-time io , and f.exptime > nbr-refresh-time io + 

T g +Ad. 
v. There is a block message in out-queue^ addressed to i, and f.exptime > now +T g + Ad. 
vi. There is a block message in transit from io to i with deadline £, and f.exptime > £ + T g + 3d. 
(b) f.exptime > now + T g + 3d 

4. If a block or lookup-comp message is in an out-queue then it contains a finger for io with exptime > now + 
T g + 3d 

5. If a block or lookup-comp message is in transit with deadline £ then it contains a finger for io with exptime > 

£ + T g + 2d. 

Proof. We proceed by induction on the number of steps in a following the join-ack^ . 

Base: steps. 

Then the last step of a is join-ack io . All the conditions are easy to check. 

Inductive step: The only actions that could falsify any of the claims are receive(lookup), send(lookup-comp), receive(lookup-comp), 

join-ping, send(ping)*, , receive(ping)*, , send(block), receive(block), join-ack, neighbor-refresh, v, and garbage-collect. 

We consider cases. 

1. receive(lookup)* i ,. 

This has the potential to falsify Property 4, in the case where a lookup-comp message is placed in out-queue i . 
By inductive hypothesis, Property 3(b), in the pre-state of the final transition, there exists / € fingers i such that 
f.phys = io and f.exptime > now + T g + 3d Therefore, if a lookup-comp message is placed in out-queuei 
as a result of this transition, it contains a finger for io with exptime > now +T g + 3d This shows Property 4. 

2. send(lookup-comp),j 

This could falsify Property 5. In the pre-state of the final transition, a lookup-comp message is in out-queuei. 
Therefore, by inductive hypothesis, Property 4, this message contains a finger for io with exptime > now + 
T g + 3d Since t < now + d, we have exptime > £ + T g + 2d, as needed for Property 5. 

3. receive(lookup-comp)*,. 

This could falsify Property 1. Before the step, a lookup-comp message is in transit to i with deadline > now. By 
inductive hypothesis, Property 5, this message contains a finger for io with exptime > now +T g + 2d. So after 
the step, fingerSi contains a finger / for i with f.exptime > now + T g + 2d. Since ping-timei < now + T g , 
we have that f.exptime > ping-timei + 2d This shows both parts of Property 1. 

4. join-ping^. 

This could falsify Property 1(a) or 2(a). For Property 1(a), suppose that statusi = joining and a contains a 
receive(lookup-comp)*j event for target PToX(i). The interesting case is where la(i) is true just before the 
step, that is, fingers t contains a finger / for Iq with f.exptime > ping-timei + 2d Since ping-timei ^ now, 
this implies that f.exptime > now + 2d. This inequality is true after the step as well. 

We claim that the step results in a ping message addressed to io being placed in out-queuei', ^ s means tnat 
la(ii) is satisfied in the post-state, as needed. Since we have assumed that Property l(a)i is true in the pre-state, 
we know that fingerSi contains a finger for io in the pre-state. Since a receive(lookup-comp) occurs in a for 
target PToX(i), we know that there exists r such that r. id £ join.compi and r. target = PToX(i). Therefore, 
the join-ping deposits ping messages addressed to its entire c-block, according to its local ring. This includes io, 
as needed. 

For Property 2(a), the argument is similar to that for Property 1(a). This time, suppose that statusi = joining 
and a contains a receive(block)*j event. The interesting case is where 2a(i) is true just before the step, that is, 
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fingers i contains a finger / for i with f.exptime > ping-time^ + 2T g + 7d. Since ping-time i > now, this 
implies that /. exptime > now + 2T g + Id. This inequality is true after the step as well. 

We claim that the step results in a ping message addressed to io being placed in out-queue^, this means that 
2(a)ii is satisfied in the post-state, as needed. Since we have assumed that Property 2(a)i is true in the pre-state, 
we know that fingers i contains a finger for io in the pre-state. Since a receive(lookup-comp) occurs in a for 
target PToX(i), we know that there exists r such that r.id G join.comp i and r. target = PToX(i). Therefore, 
the join-ping deposits ping messages addressed to its entire c-block, according to its local ring. This includes io, 
as needed. 

5. send(ping) Mo . 

This could falsify Property 1(a) or 2(a). For Property 1(a), suppose that statusi = joining and a contains a 
receive(lookup-comp)*j event for target PToX(i). The interesting case is where la(ii) is true just before the 
step, that is, fingers i contains a finger / for io with f.exptime > now + 2d and there is a ping message in 
out-queue i addressed to io- After the step, there is a ping message in transit from i to io with deadline now + d. 
Taking I = now + d, we see that lc is true after the step. 

For Property 2(a), the argument is similar: 2(a)ii before the step implies 2(a)iii after the step. 

6. receive(ping) Mo . 

This could falsify Property 1(a) or 2(a). For Property 1(a), suppose that statusi = joining and a contains a 
receive(lookup-comp)*j event for target PToX(i). The interesting case is where la(iii) is true just before the 
step, that is, fingers i contains a finger / for io with f.exptime > I + d and there is a ping message in transit 
from i to io with deadline I. Since I > now, we have that f.exptime > now + d. After the step, there is a 
block message in out-queue i addressed to i. Therefore, la(iv) is true just after the step. 

For Property 2(a), the argument is similar: 2a(iii) before the step implies 2a(iv) after the step. 

7. send(block)j^. 

This could falsify Property 1(a), 2(a), 3(a), or 5. For Property 1(a), the interesting case is where j = io, k = i, 
and l(a)iv is true before the step, that is, fingers i contains a finger / for io with f.exptime > now + d. Since 
now + d>l,we have that f.exptime > I, so that la(v) holds after the step. 

For Property 2(a), the interesting case is where j = io, k = i, and 2(a)iv holds before the step. Then, arguing as 
in the previous case, 2(a)v holds after the step. 

For Property 3(a), there are two interesting cases. The first is where j = i, k = io, and 3(a)ii holds before the 
step; in this case 3(a)iii holds after the step. The second case is where j = io, k = i, and 3(a)v holds before the 
step; in this case 3(a)vi holds after the step. 

For Property 5, we use Property 4 in the pre-state to show Property 5 in the post-state. 

8. receive(block)j^ 

This could falsify Property 1(a), 2(a), or 3(a). 

For Property 1(a), the interesting case is where j = io, k = i, and Property l(a)v holds before the step. Then 
by Lemma 6.5, the received message contains a finger for io with exptime > now +T e — d. By assumptions 
on the constants, the right-hand side is > AT g + Id, so exptime > now + AT g + Id. Therefore, after the step, 
fingers { contains a finger for i with exptime > now + 4T g + Id > ping-time i + 2d. Thus, l(a)i is satisfied 
after the step. 

For Property 2(a), the interesting case is where j = io, k = i, and Property 2(a)v holds before the step. Arguing 
as in the previous case, we see that after the step, fingers i contains a finger for io with exptime > AT g + Id > 
ping-time i + 2T g + Id. Thus, 2(a)i is satisfied after the step. 

For Property 3(a), there are two interesting cases. The first is where j = i, k = io, and 3(a)iii is satisfied before 
the step; then we claim that 3(a)iv holds after the step. The argument for this uses Lemma 6.5, applied to i. The 
second case is where j = io, k = i, and 3a(vi) is satisfied before the step; in this case, 3(a)i holds after the step. 
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9. join-ack^. 

This could falsify Property 3(a). By inductive hypothesis, Property 2(b), in the pre-state, fingers i contains a 
finger for i with exptime > now + 2T g + 5d. Since nbr-refresh-time i = now right after the step, 3(a)i holds 
after the step. 

10. neighbor-refresh^. 

This could falsify Property 3(a). The interesting case is where Property 3a(i) holds in the pre-state. The step 
puts a block message in out-queue i addressed to iq. Then 3a(ii) holds in the post-state. 

11. v{t) 

This could falsify Property 1, 2, 3, or 4. For Property 1, there are two interesting cases. The first is where l(a)iv 
holds in the pre-state. But then time cannot pass, by our timing assumption (no time passes while an out-queue 
is nonempty). The second possibility is that we might falsify 1(b). However, note that 1(b) follows from 1(a). 
Similar arguments hold for Properties 2, 3, and 4. 

12. garbage-collect. 

Since in every case, the finger whose existence is claimed has exptime > now, it cannot be garbage-collected. 
Therefore, garbage-collect cannot falsify any of the claims. 

□ 

Next, we describe knowledge that iq acquires about the other processes. 

Lemma 6.7. Let a be a good finite execution that contains no fail events, and contains at least one and at most 2c + 1 
join-ack events. Let {q denote the process that performs the first join-ack in a. Let i € PId be such that join-ack^ 
occurs in a at time t. 
Then in £state(a), one of the following holds: 

1. t = now and a block message addressed to iq is in out-queue^ 

2. A block message is in transit from i to i® with deadline t + d. 

3. fingers^ contains a finger f for i such that one of the following holds: 

(a) f. exptime > nbr-refresh-timei + T e — T g . 

(b) A block message addressed to i® is in out-queue i and f .exptime > now + T e — T g . 

(c) A block message is in transit from i to iq with deadline £ and f .exptime > £ + T e — (T g + d). 

Proof. By induction on the number of steps in a following the join-ack^. 

Base: steps. 

Then the last step of a is join-ack^. Then we claim that Property 1 holds in the post-state. This follows because in the 

pre-state, i has a finger for Iq, by Lemma 6.6, part 3(b). 

Inductive step: The only actions that could falsify the claim are send(block)j, receive(block)j , neighbor-refresh,, 

time-passage, and garbage-collect^ . 

1. send(block). 

This could falsify Property 1 or 3(b). However, if it does so, it makes Property 2 or 3(c) (respectively) true. 

2. receive(block), 

Lemma 6.5 implies that after the step, fingers io contains a finger for i with exptime > I + T e — d, where £ 
is the deadline component of the received message. Since £ = nbr-refresh-time i — T g + d, (the sending time 
plus d), this implies that this finger has exptime > nbr-refresh-timei — T g + d + T e — d, that is, exptime > 
nbr-refresh-timei +T e — T g , which shows that 3(a) is satisfied after the step. 

3. neighbor-refresh^ 

This could falsify Property 3(a); however, if it does so then Property 3(b) holds after the step. 
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4. v{i) 

This could falsify Property 1 or 3(b). However, if 1 or 3(b) holds in the pre-state, then time cannot pass, by our 
timing assumptions, because an out-queue is nonempty. 

5. garbage-collect^. 

Because T e > T g + d, the expiration times of the claimed fingers are all strictly greater than 0. Therefore, this 
cannot falsify any of the statements. 

□ 

The following corollary summarizes the conclusions of Lemma 6.7, saying that io has a finger for any other process i 
that has joined at least time d ago, with a high expiration time. Also, any block message that is sent by i(, sufficiently 
long after i joins contains a finger for i with a high expiration time. 

Corollary 6.8. Let a be a good finite execution that contains no fail events, and contains at least one and at most 
2c + 1 join-ack events. Let iq denote the process that performs the first join-ack in a. Let i (E PId be such that 
join-ack^ occurs in a at time t. 
Then in £state(a), the following hold: 

1. Ift + d < now then fingers^ contains a finger f for i such that f.exptime > now +T e — (T 9 + d). 

2. Ift + 2d < £ and a block message is in transit from i$ with deadline £, then the message contains a finger for i 
such that f.exptime > £ + T e — (T g + 2d). 

The next lemma gives guarantees about what an arbitrary process i knows about another arbitrary process j. This 
represents "second-order" information, because i may need to learn this information indirectly, through iq. 

Lemma 6.9. Let a be a good finite execution that contains no fail events, and contains at least one and at most 2c + 1 
join-ack events. Let i® denote the process that performs the first join-ack in a. Let s = £state(a). Then: 

1. Suppose that s.statusi = joining and a contains a receive(block)j j event. Suppose that join-ack ■ occurs in a 
at a time < £time(a) — (T g + 3d). 

Then s.fingers i contains a finger f for j such that f.exptime > s.now +T e — (2T g + 3d). 

2. Suppose that s.statusi = active and join-ack^ occurs in a at a time > £time(a) — (T g + 2d). Suppose that 
join-ack- occurs in a at a time < £time(a) — (2T g + 5d). Then s.fingers i contains a finger f for j such that 
f.exptime > s.now + T e — (3T g + 5d). 

3. Suppose that s.statusi = active and join-ack^ occurs in a at a time < £time(a) — (T g + 2d). Suppose that 
join-ack- occurs in a at a time < £time(a) — (T g + 2d). Then s.fingers i contains a finger f for j such that 
f.exptime > s.now +T e - (2T g + 2d). 

The proofs are based on conveying information through i^. These proofs are not inductive; rather, they rest directly on 
previously-proved lemmas. 

Proof. 1. Assume that s.statusi = joining and a contains a receive(block)j j event. Also suppose that join-ack 

occurs in a at a time < £time(a) — (T g + 3d). 

Lemma 6.6, Part 1(b), implies that whenever i sends a ping message during its joining protocol, it has a finger 
for iq. Thus, by the limitation on the number of join-ack events, i(, is included in the set of destinations of the 
ping message. 

We claim that, in a, process i receives a block message from i(, sent by i(, in response to a ping message sent by 
i at a time > s.now — (T g + 2d) . For if not, then the latest block message received by i from i(, is a response to 
a ping sent by i at a time < s.now — (T g + 2d). But then it must be that another ping message is sent by i at a 
time < s.now — 2d, and this receives a response by the end of a, a contradiction. 

Since the time of the join-ack is < s.now — (T g +3d), it must be < s 1 .now — d, where s 1 is the state just before 
id sends this block message. Therefore, by Corollary 6.8, Part 1, in state s 1 , fingers io contains a finger for j 
with exptime > s 1 .now + T e — (T g + d). Therefore, in state s, which is at most time T g + 2d later, fingers i 
contains a finger for j with exptime > s.now + T e — (2T g + 3d), as needed. 
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2. Suppose that s.statusi = active and join-ack^ occurs in a at a time > ltime(a) — (T g + 2d). Also suppose that 
join-ack^ occurs in a at a time < £time(a) — (2T g + 5d). 

By the inductive hypothesis, Part 1, we know that, just before the join-ack^, fingers i contains a finger for j with 
exptime > now +T e — (2T g + 3d). Therefore, in state s, which is at most time T g + 2d later, fingers i contains 
a finger for j with exptime > s.now + T e — (3T g + 5d), as needed. 

3. Suppose that s.statusi = active and join-ack^ occurs in a at a time < ltime(a) — (T g + 2d). Suppose that 
join-ack^ occurs in a at a time < ttime{a) — (T g + 2d). 

Corollary 6.8, Part 1, implies that in any state s 1 of a with s 1 .now > t' + d, fingers i contains a finger for i 
with exptime > s 1 .now +T e — (T g + d). Since T e > T g + d and because of the limitation on the number of 
join-ack events, i is included in the set of destinations of every block message sent by i(, in such a state s'. 

We claim that, in a, process i receives a block message from io sent by io at a time > s.now — (T g + d). For 
if not, then the latest block message received by i from io is sent by io at a time < s.now — (T g + d). But then 
it must be that another block message is sent by io (as part of a neighbor-refresh io ) at a time < s.now — d, and 
this arrives at i by the end of a, a contradiction. 

Now fix s 1 to be the state just before io sends this block message; thus, s 1 .now > s.now — (T g + d). Putting 
this inequality together with the assumption that the join-ack occurs at a time < s.now — (T g + 2d), we may 
conclude that the join-ack occurs at a time < s 1 .now — d. Therefore, by Corollary 6.8, Part 1, in state s', 
fingers io contains a finger for j with exptime > s 1 .now + T e — (T g + d) . Therefore, in state s, which is at most 
time T g + d later, fingers { contains a finger for j with exptime > s.now + T e — (2T g + 2d), as needed. 

□ 

The following lemma describes information that i is guaranteed to have after receiving a lookup-comp message. It rep- 
resents "third-order" information, because the lookup-comp message could be conveying "second-order" information 
from its sender. 

Lemma 6.10. Let a be a good finite execution that contains no fail events, and contains at least one and at most 2c +1 
join-ack events. Let i,j € PId. Suppose that statusi = joining and a contains a receive(lookup-comp)*j event for 
target PToX(i). Suppose that join-ack^ occurs in a, at a time < Itime(a) — (3T fl + 8d). 
Then in Istate(a), fingers i contains a finger for j with exptime > now. 

Proof. (Sketch:) If the time when process i receives the lookup-comp message is < ltime(a) — (T g + 2d), then i 
also receives a block message from io before the end of a. In this case the result follows from Lemma 6.9, Part 1. 

On the other hand, if the time when process i receives the lookup-comp message is > ltime(a) — (T g + 2d), then 
the result follows from Lemma 6.9, part 2, applied to the sender of the message. In applying this lemma, we add time 
T g + 3d (d for the message delay and T g + 2d for the time that might have elapsed from the receive(lookup-comp)) 
to the age of the known processes and subtract this from the expiration time of the finger. This uses the fact that 
T e > AT g + 9d. D 

The next series of results bound how long it takes for a process i to become an "authority", like io- That is, it knows 
about all processes that have joined more than time d ago. The first case is where another process j joins sufficiently 
long after i so that j knows about i at the point where it joins. 

Lemma 6.11. Let a be a good finite execution that contains no fail events, and contains at least one and at most 
2c + 1 join-ack events. Suppose that join-ack^ and join-ack occur in a at times t and t', respectively, and where 
t + T g + 3d<t' < now - d. 
Then in Istate(a), fingers i contains a finger for j with exptime > now +T e — (T g + d). 

Proof. We first claim that, at any point in a after the join-ack , fingers j contains a finger fori with exptime > now. 
Lemma 6.9, Part 1, implies that, in the state immediately before the join-ack p fingers j contains a finger for i with 
exptime > now + T e — (2T g + Ad). Thereafter in a, through time t 1 +T g + 2d, fingers j contains a finger for i with 
exptime > now + T e — (3T g + 6d). Also, at any time after t' + T g + 2d in a, Lemma 6.9, Part 2 implies that fingers j 
contains a finger for i with exptime > now + T e — (3T g + 6d). Combining these two facts, we conclude that, at any 
time after the join-ack^, fingers j contains a finger for i with exptime > now + T e — (3T g + 6d) > now. 
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Immediately after the join-ack , and at intervals of T g thereafter, process j performs a neighbor-refresh , in which 
it sends a block message containing a finger for itself with exptime = T e . By the argument in the previous paragraph, 
i is included in the destination set of each such block message. At the end of a, some such message must have arrived 
at i which was sent by j at a time > ltime(a) — (T g + d). Therefore, in lstate(a), fingers i contains a finger for j 
with exptime > now + T e — (T g + 2d), as needed. □ 

The second case is where i and j both join long enough before the end of the execution. 

Lemma 6.12. Let a be a good finite execution that contains no fail events, and contains at least one and at most 
2c + 1 join-ack events. Suppose that join-ack^ and join-ack occur in a at times t and t', respectively, where t, t' < 
Itime(a) - (2T g + M). 
Then in Istate(a), fingers t contains a finger for j with exptime > now +T e — (T g + d). 

Proof. By Corollary 6.8, Part 1, by time strictly less than £time(a) — (2T g + 2d), fingers io contains fingers for 
both i and j, each with exptime > now + T e — (T g + 2d). Then by time strictly less than ltime(a) — (T g + d), 
j receives a block message from i(, telling j about i, resulting in fingers j containing a finger for i, with exptime > 
now + T e — (2T g + 3d). And then by time strictly less than ltime(a), i receives a block message directly from j 
telling i about j, and producing the needed finger. □ 

The following corollary says that if process i has joined more than time 3T g + 6d ago, it is an "authority", in the sense 
that it knows about all processes j that has joined more than time d ago. 

Corollary 6.13. Let a be a good finite execution that contains no fail events, and contains at least one and at most 
2c + 1 join-ack events. Suppose that join-ack^ and join-ack occur in a at times t and t', respectively, where t < 
£time(a) — (3T fl + 6d) andt' < Itime(a) — d. 
Then in Istate(a), fingers i contains a finger for j with exptime > now + T e — (T g + d). 

Proof. This follows from the two previous lemmas. □ 

6.4.3 Joins and failures 

Now we use the ideas in the previous section to talk about what happens when we have unlimited joins and also 
failures. Now, instead of relying on i(, as an "authority", processes rely on neighbors that happen to have been around 
long enough. Because of the failures, we now consider the augmented ring as well as the actual global ring. 

From now on, I am being slightly sloppy by writing just i instead of PToX (i) in many places. This is done for the 
sake of readability. I hope it does not cause any confusion. The first lemma relates various neighborhoods in the same 
ring. 

Lemma 6.14. Let R be any ring, i,j,k € PId. 

1. If j G block(i,ei,R) andk € block(i,e2,R), then j € block(k,ei +e2,R). 

2. Ifj £ succset(i,ei,R), k € succset(i,e2,R), andk fi succset(i,e^,R),thenj € block (k,m&x(ei — e^,e2),R). 

Proof. Straightforward. □ 

The following lemma asserts the existence of neighbors that have joined a long time ago. 

Lemma 6.15. Assume that c > e\ + e^ + ezjoinbd. Let a be a good finite execution, R = global-ring (a). Let 
i € PId. Suppose that \R\ > c + 1. Then: 

1. There exists k € PId such that 

(a) k £ succset(i, c — e\ , R). 

(b) k ^ succset(i,e2,R). 

(c) join-ack^. occurs at a time < Itime(a) — e^(T g + 2d) 

(d) failfc does not occur in a. 
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2. There exists k £ PId such that 

(a) k £ predset(i, c — e\ , R). 

(b) k (£ predset(i,e2,R). 

(c) join-ackj, occurs at a time < Itime(a) — e^(T g + 2d) 

(d) failfc does not occur in a. 

Proof. We prove Part 1; Part 2 is analogous. There are at least c — (ei + e-i) processes in the set difference 
succset(i,c — e\,R) — succset(i,e2,R). Of these, at most e^joinbd perform a join-ack at times > £time(a) — 
e s(T g + 2d). Since c > e\ + ei + ezjoinbd, it must be that at least one of these processes, call it k, performs a join-ack 
at a time < ltime(a) — ez(T g + 2d). This k satisfies all the listed properties. □ 

The next lemma relates neighborhoods in the global ring to neighborhoods in the augmented ring. 
Lemma 6.16. Let a be a good finite execution, e £ N, i, j £ PId. 

1. If j G psuccset(i,e, global-ring (a)) then j € psuccset(i,e +failbd, aug-ring(a)). 

2. If j £ ppredset(i,e, global-ring (a)) then j € ppredset(i,e +failbd, aug-ring(a)). 

3. If j € succset(i,e, global-ring (a)) then j € succset(i, e + failbd , aug-ring(a)). 

4. If j £ predset(i, e, global-ring (a)) then j £ predset(i, e + failbd, aug-ring(a)). 

5. If j £ block(i,e, global-ring (a)) then j £ block(i,e + failbd, aug-ring(a)). 

Proof. (Sketch) These follow because at most failbd processes in the given region appear in aug-ring(a) but not in 
global-ring (a). □ 

The next lemma says that neighbors in the augmented ring are also neighbors in the local ring. 
Lemma 6.17. Let a be a good finite execution, s = £state(a). 

1. If j £ succset(i,e, aug-ring(a)) and fingers i contains a finger for j, then j £ succset(i,e,s. local-ring ,). 

2. Ifj £ predset(i, e, aug-ring(a)) and fingers i contains a finger for j, then j £ predset(i, e, s. local-ring^. 

3. If j £ block(i,e, aug-ring(a)) and fingers i contains a finger for j , then j £ block (i,e,s. local-ring ,). 

Proof. We show Part 1; the rest are similar. Ifj ^ succset(i, e, s. local-ring^, then it must be that there are at least 
e elements of s .local-ring J in the interval (i, j). But each of these is an element of aug-ring(a)), which contradicts 
the assumption that j £ succset(i, e, aug-ring(a)). □ 

The next lemma relates the augmented ring at some point to the global ring at a point not too far in the past. 

Lemma 6.18. Let a be a good finite execution, a 1 aprefixofawithltime(a') > ltime(a)—T e . Ifi £ global-ring (a 1 ), 
theni £ aug-ring(a). 

Proof. By the definition of aug-ring. □ 

The next lemma says that a neighbor in the augmented ring at a particular time is a neighbor in the global ring at a 
point not too far in the past. 

Lemma 6.19. Let a be a good finite execution, a' a prefix of a with £time(a') > £time(a) — T e . Let e £ N and 
i,j £ PId. Suppose j £ global-ring (a 1 ). Then: 

1. Ifj £ psuccset(i,e, aug-ring(a)) then j £ psuccset(i,e, global-ring (a 1 )). 

2. Ifj £ ppredset(i,e, aug-ring(a)) then j £ ppredset(i,e, global-ring (a 1 )). 
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3. If j £ succset(i,e, aug-ring(a)) then j € predset(i,e, global-ring (a 1 )). 

4. If j £ predset(i, e, aug-ring(a)) then j € predset(i, e, global-ring(a')). 

5. If 'j € block(i,e, aug-ring(a)) then j € block(i,e, global-ring (a 1 )). 

Proof. For Part 1, suppose for the sake of contradiction that j ^ psuccset(i, e, global- ring (a 1 )). Then \ global- ring (a' )tl 
(i,j)\ > e, that is, there are more than e elements of global-ring (a 1 ) in the interval properly between i and j, 
moving in the clockwise direction. By Lemma 6.18, every such element is also in aug-ring(a'). Therefore, j ^ 
psuccset(i,e, aug-ring(a)). This is a contradiction. 

The proof of Part2is analogous. For Part 3, suppose that j € succset(i,e, aug-ring(a)). If j € psuccset(i,e, aug-ring(a)) 
then the conclusion follows from Part 1. The only remaining case is where j = i, but this case follows trivially from 
the fact that j g global- ring (a 1 ). 

Part 4 is analogous. Part 5 follows from Parts 3 and 4. □ 

The following lemma summarizes facts about the knowledge of a new process at various points during and soon after 
its joining protocol. 

Lemma 6.20. Let a be a good finite execution, s = £state(a). Let i be a process that does not fail in a. Then: 

1. Suppose that s.statusi = joining and a receive(lookup-comp)*j event for target i occurs in a at a time > 
Itime(a) — (T g + 2d). Suppose that j € block(i, c, aug-ring(a)), join-ack^ occurs in aata time < Itime(a) — 
(3T g + 8d), and failj does not occur in a. 

Then fingers i contains a finger for j with exptime > now. 

2. Suppose that statusi = joining and a receive(lookup-comp)*j event for target i occurs in a at a time < 
£time(a) — (T g +2d). Suppose that j € block(i, b, aug-ring(a)), join-ack^ occurs in aata time < Itime(a) — 
(T g + 3d), and failj does not occur in a. 

Then fingers i contains a finger for j with exptime > now + T e — (2T fl + 3d). 

3. Suppose that s.statusi = active and a join-ack^ occurs in a at a time > £time(a) — (T g + 2d). Suppose that 
j € block(i, b, aug-ring(a)), join-ack^ occurs in aata time < Itime(a) — (2T g + 5d), andfaWj does not occur 
in a. 

Then s.fingers i contains a finger for j with exptime > s.now + T e — (3T fl + 6d). 

4. Suppose that s.statusi = active and a join-ack^ occurs in a at a time < £time(a) — (T g + 2d). Suppose that 
j € block(i, b, aug-ring(a)), join-ack^ occurs in aata time < Itime(a) — (T g + 2d), and f a i I _,- does not occur 
in a. 

Then s.fingers i contains a finger for j with exptime > s.now +T e — (2T g + 2d). 

5. Suppose that s.statusi = active and a join-ack^ occurs in a at a time < Itime(a) — (3T g + 6d). Suppose that 
j € block(i,b — failbd, aug-ring(a)), join-ack^ occurs in a at a time < Itime(a) — d, andfaWj does not occur 
in a. 

Then s.fingers i contains a finger for j with exptime > s.now + T e — (T g + d). 

Proof. Let R denote global-ring (a). The proof is by strong induction on the number of steps in a. 
Base: The total number of join-ack events in a is at most 2c + 1. 

If there are no join-ack events in a then the statements are all vacuously true. If there are between one and 2c + 1 
join-ack events in a then the five claims follow from Lemma 6.10, Lemma 6.9, Parts 1, 2, and 3, and Corollary 6.13, 
respectively. (This uses the fact that, in the absence of failures, aug-ring is the same as global-ring .) 
Inductive step: We assume that a contains more than 2c + 1 join-ack events. We assume that the result is true for all 
proper prefixes of a and show it for a. We show the five properties in turn. 

1. For Part 1, suppose that s.statusi = joining, a receive(lookup-comp)*j event for target i occurs in a at a time 
> ttime{a) — (T g + 2d), and fail& does not occur in a. Also suppose that j € block(i, c, aug-ring(a)) and 
join-ack^ occurs at a time < ttime{a) — (3T g + 8d). We must show that s.fingers i contains a finger for j with 
positive exptime. 
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Consider the first lookup-comp message for target i that is received by i, and let A; be the sender of this message. 
Let a 1 be the prefix of a ending just before the receive(response) step in which k sends this message, let 
s 1 = £state(a') and let R 1 = global- ring (a 1 ). 

By inductive hypothesis, Parts 3 and 4, s 1 . fingers k contains a finger for every process in succset(k, b, aug-ring(a')) 
whose join-ack event occurs at a time < ltime(a') — (2T g + 5d) and that does not fail in a! . Therefore, by 
Lemma 6.16, s' .fingers k contains a finger for every process in succset(k, b — failbd, R 1 ) whose join-ack event 
occurs at a time < £time(a') — (2T g + 5d). In particular, s' .fingers k contains a finger for every process in 
succset(k, min (\R' (~l [k,i)\,b — failbd), R 1 ) whose join-ack event occurs at a time < £time(a') — (2T g + 5d). 

By our assumption on the join rate, at most 3joinbd processes in succset(k, b — failbd, R 1 ) perform join-ack 
events at times > £time(a') — (2T g +5d). It follows that s 1 .fingers k contains at least min (\R' (~l [k,i)\,b — failbd) — 
Sjoinbd fingers for processes in R 1 (~l [k, i). 

Now we claim that k € ppredset(i,c + 3joinbd,R'). If not, then \R' fl [k,i)\ > c + 3joinbd. Then, 
since b > c + 3joinbd + failbd, we have that min(|i?' n [k , i)\ , b — failbd) — 3joinbd > c, which im- 
plies that s 1 .fingers k contains strictly more than c fingers for processes in R 1 (~l [k,i). This implies that 
k (£ ppredset(i,c,s' .local-ring ,), However, the definition of the receive(response) transitions implies that 
k £ ppredset(i, c, s 1 .local-ring^, which yields a contradiction. Therefore, k £ ppredset(i, c + 3joinbd, if"), as 
claimed. 

Since j € block(i,c, aug-ring(a)), Lemma 6.19 implies that j € block(i,c,R'). Since j € block(i,c,R') and 
k £ predset(i, c + 3joinbd,R'), Lemma 6. 14 implies that j € block(k,2c + 3joinbd, R 1 ). Since (by assumption 
on constants) b > 2c + 3joinbd + failbd, we have that j € block(k, b — failbd, R 1 ). Therefore, by Lemma 6.16, 
j € block(k,b,aug-ring(a')). 

Now we use the inductive hypothesis, Parts 3 and 4, again, to conclude that s 1 .fingers k contains a finger for j 
with exptime > s 1 .now +T e — (3T g + 6d), To apply the inductive hypothesis, we need the fact that join-ack 
occurs at a time < ltime(a') — (2T g + 5d); this follows from our assumption that join-ack- occurs at a time 

< ttimeia) - (3T g + 8d) and the fact that £time(a') > £time(a) - (T g +3d). 

Since j € block(k,b, aug-ring(a')) and s 1 .fingers k contains a finger for j, Lemma 6.17 implies that j € 
block(k, b, s 1 .local-ring k ). Therefore, this finger for j gets included in the block sent by k in the lookup-comp 

message. 

Upon receipt of this message, fingers i contains a finger for j with exptime > now +T e — (3T g + 7d). Then at 
the end of a, at most time T g + 2d later, fingers i contains a finger for j with exptime > s.now+T e — (4T fl + 9d) . 
Since T e > AT g + 9d, this implies exptime > s.now, as needed. 

2. For Part 2, suppose that statusi = joining and a receive(lookup-comp)*j event for target i occurs in a at a 
time < £time(a) — (T g + 2d). Suppose that j € block(i,b, aug-ring(a)), join-ack^ occurs in a at a time 

< ltime(a) — (T g + 3d), and failj does not occur in a. We must show that fingers i contains a finger for j with 
exptime > now + T e — (2T g + 3d). Without loss of generality, assume that j € succset(i, b, aug-ring(a)). 

We first claim that there exists k € PId such that k € succset(i,c — 2failbd,R) — succset(i,2failbd,R), 
join-ack^. occurs at a time < ltime(a) — (4T fl + lOd), and fa\\ k does not occur in a. This follows from 
Lemma 6.15, applied with e\ = ei = 2joinbd and e% = 5, using the assumption that c > hjoinbd + 4failbd. 

Now we claim that j € block (k,b— 2 failbd, aug-ring(a)). We know that A; € succset(i,c— failbd, aug-ring(a))). 
Also, since A; ^ succset(i, 2failbd, R), we have that k ^ succset(i,2failbd,aug-ring(a)). Also, by assump- 
tion, j € succset(i,b, aug-ring(a)). Lemma 6.14, Part 2, applied with e\ = c— failbd, e^ = 6ande3 = 2failbd, 
then implies that j € block(k, b — 2failbd, aug-ring(a)), as claimed. 

Process i performs ajoin-ping at some time in the left-closed, right-open interval [ltime(a) — (T g +2d) , ltime(a) — 
2d), and i receives responses for all ping messages generated by that join-ping whose destinations do not fail. 
Let a 1 be the prefix of a ending just before the join-ping^, s 1 = £state(a'), and R 1 = global-ring(a'). 

We claim that s' .fingers { contains a finger for k. Since the time of the join-ack^. is < ltime(a) — (4T fl + lOd), 
it is also < itime(a') — (3T g + 8d). Since k € succset(i,c — 2failbd,R), Lemma 6.16 implies that k € 
succset(i, c — failbd, aug-ring(a). Therefore, by Lemma 6.19, k € succset(i, c — failbd, R 1 ). Therefore, by 
Lemma 6.16, k € succset(i, c, aug-ring(a')). Then the inductive hypothesis, Part 1, implies that s' .fingers { 
contains a finger for k. 
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Since k € succset(i,c, aug-ring(a')) and s' '.fingers i contains a finger for k, Lemma 6.17 implies that k € 
block(i, c, s 1 .local-ring i ). Therefore, during the join-ping, i sends a ping message to k. Since k does not fail in 
a, k responds to the ping message with a block message. Let a" be the prefix of a ending just before k sends 
the block message, let s" = £state(a"), and let R" denote global-ring (a"). 

Since j € block(k, b — 2failbd, aug-ring(a)), Lemma 6.19 implies that j € block(k, b — 2failbd, R"). Then 
Lemma 6.16 implies that j € block(k,b — failbd, aug-ring(a")). Then by inductive hypothesis, Part 5, we 
know that s" . fingers k contains a finger for j with exptime > s" .now + T e — (T g + d). 

Since j € block(k,b, aug-ring(a")) and s" .fingers k contains a finger for j, Lemma 6.17 implies that j € 
block(k, b, s" .local-ring k ) . Therefore, the finger for j is included in the block sent by k in its block message to 
i. At most T g + 2d time elapses from this send until the end of a, which means that s. fingers i contains a finger 
for j with exptime > s.now + T e — (2T g + 3d), as needed. 

For Part 3, suppose that s.statusi = active and a join-ack^ occurs in a at a time t > £time(a) — (T g + 2d). 
Suppose also that j € block(i, b, aug-ring(a)), join-ack^- occurs at a time < £time(a) — (2T g + 5d), and failj 
does not occur in a. We must show that s. fingers i contains a finger for j with exptime > s.now +T e — (3T fl + 
6d), Without loss of generality, assume that j € succset(i, b, aug-ring(a)). 

The argument is similar to that for the previous case, because we argue with respect to pings and block re- 
sponses near the end of the joining protocol. By Lemma 6. 15, there exists k € PId such that k £ succset(i,c — 
2failbd,R) — succset(i,2failbd, R), join-ackj. occurs at a time < £time(a) — (5T g + 12d), and fa\\ k does 
not occur in a. This uses the assumption that c > Qjoinbd + Afailbd. Then, since k € succset(i,c — 
failbd, aug-ring(a)), k (£ succset(i, 2failbd, aug-ring(a)), and j € succset(i, b, aug-ring(a))), Lemma 6.14, 
Part 2, implies that j € block(k, b — 2failbd, aug-ring(a)). 

Process i performs a join-ping at some time in the interval [t — (T g + 2d),t — 2d), and i receives responses 
for all ping messages generated by that join-ping whose destinations do not fail in a. Let a 1 be the prefix of a 
ending just before the join-ping^, s 1 = £state(a'), and R 1 = global-ring (a 1 ). 

We claim that s 1 .fingers i contains a finger for A;. Since the time of the join-ackj. is < £time(a) — (5T g + 12d), 
it is also < ltime{a!) — (3T g + 8d). Since k is in succset(i, c — 2failbd, R), Lemma 6.16 implies that k € 
succset(i,c — failbd, aug-ring(a). Therefore, by Lemma 6.19, A; € succset(i,c — failbd, R 1 ). (This uses 
the assumption that T e > 2T g + Ad.) Therefore, by Lemma 6.16, k € succset(i, c, aug-ring(a')). Then the 
inductive hypothesis, Parts 1 and 2, imply that s' .fingers i contains a finger for k. 

Since k € succset(i,c, aug-ring(a')) and s' .fingers { contains a finger for k, Lemma 6.17 implies that k € 
block(i, c, s 1 .local-ring i ). Therefore, during the join-ping, i sends a ping message to k. Since k does not fail in 
a, k responds to the ping message with a block message. Let a" be the prefix of a ending just before k sends 
the block message, let s" = £state(a"), and let R" denote global-ring (a"). 

Since j € block(k, b — 2failbd, aug-ring(a)), Lemma 6.19 implies that j € block(k, b — 2failbd, R"). Then 
Lemma 6.16 implies that j € block(k,b — failbd, aug-ring(a")). Then by inductive hypothesis, Part 5, we 
know that s" .fingers k contains a finger for j with exptime > s" .now + T e — (T g + d). (Here, we need the fact 
that the time of the join-ackj. is < £time(a") — (3T g + 6d), and the time of the join-ack^ is < £time(a") — d.) 

Since j € block(k,b, aug-ring(a")) and s" .fingers k contains a finger for j, Lemma 6.17 implies that j € 
block(k, b, s" .local-ring i ). Therefore, the finger for j is included in the block sent by k in its block message 
to i. At most 2T g + Ad time elapses from this send until the end of a, which means that s.fingers i contains a 
finger for j with exptime > s.now + T e — (3T fl + 5d), which suffices. 

For Part 4, suppose that s.statusi = active andajoin-ack^ occurs inaat atime < £time(a) — (T g +2d). Suppose 
that j e block(i, b, aug-ring(a)), join-ack^ occurs in a at a time < £time(a) — (T g + 2d), and failj does not 
occur in a. We must show that s.fingers i contains a finger for j with exptime > s.now + T e — (2T g + 2d). 
Without loss of generality, assume that j € succset(i, b, aug-ring(a)). 

Lemma 6. 15 implies that there exists A: € PId such that k € succset(i,c — 2 failbd, R) — succset(i, 2 failbd, R), 
join-ackj. occurs at a time < £time(a) — (AT g + lOd), and fa\\ k does not occur in a. This uses the assumption 
that c > 5joinbd + Afailbd. 
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At some time in the interval [£time(a) — (T g + d),£time(a) — T g ), k performs a neighbor-refresh k whose 
messages all arrive by the end of a. Let a 1 be the prefix of a ending just before this neighbor-refresh k , 
s 1 = Istate(a'), and R! = global-ring (a 1 ). 

Since k € succset(i, c — 2failbd, R), Lemma 6.16 implies that k € succset(i, c — failbd, aug-ring(a)), Also, 
since i € R, we know that i € predset(k,c — 2failbd,R) and so, by Lemma 6.16, i € predset(k,c — 
failbd, aug-ring (a)). By Lemma 6.19, i € predset(k,c — failbd, R 1 ). Therefore, by Lemma 6.16, i € 
predset(k,c, aug-ring(a')). Then by inductive hypothesis, Part 5, s' .fingers k contains a finger for i with 
exptime > Itime(a') + T e — (T g + d). 

Next, we claim that j € block(k,b—2failbd,aug-ring(a)). We know that A; € succset(i,c— failbd, aug-ring(a)). 
Also, since k ^ succset(i,2failbd,R), we know that k $. succset(i,2failbd, aug-ring (a)). Then, since 
j £ succset(i, b, aug-ring(a)), Lemma 6.14 implies that j € block(k, b — 2failbd, aug-ring(a)), as claimed. 

Therefore, by Lemma 6. 19, j € block(k, b— 2failbd,R'). So by Lemma 6.16, j € block(k,b— failbd, aug-ring(a')). 
Then by inductive hypothesis, Part 5, s 1 .fingers k contains a finger for j with exptime > ltime(a')+T e — (T g + 
d). Thus, s' .fingers k contains fingers for i and j, both with exptime > Itime(a') + T e — (T g + d). 

Since i € block(k,b, aug-ring(a')) and s' .fingers k contains a finger for i, Lemma 6.17 implies that i € 
block(k,b, s 1 .local-ring k ) . Therefore, i is among the targets of the block message sent by k during the neighbor-refresh k . 

Also, since j € block(k,b, aug-ring(a')) and s 1 .fingers k contains a finger for j, Lemma 6.17 implies that 
j £ block(k,b,s' .local-ring k ). Therefore, the finger for j is included in the block sent by A; in its block message 
to i. When the finger is sent, it has exptime > s 1 .now + T e — (T g + d). Therefore, at the end of a, which is at 
most time T g + d later, s.fingers i contains a finger for j with exptime > s.now + T e — (2T g + 2d), as needed. 

For Part 5, suppose that s. status , = active and ajoin-ack^ occurs in a at a time < ltime(a) — (3T g + 6d). Also 
suppose that j € block(i, b — failbd, aug-ring(a)), join-ack^ occurs in a at a time < ttime{a) — d, and failj 
does not occur in a. We must show that s. fingers i contains a ringer for j with exptime > s .now +T e — (T g +d) . 
Without loss of generality, assume that j € succset(i, b — failbd, aug-ring(a)). Let t denote the time of the 
join-ack . We consider two cases: 

(a) t < ttimeia) - (2T g + 3d). 

Lemma 6. 15 implies that there exists A; € PId such that k 6 succset(i,c—2failbd,R) — succset(i,2failbd,R), 
join-ackj. occurs at a time < ltime(a) — (5T g + 8d), and fail^ occurs in a. Then (as in the argu- 
ment for Part 2), Lemma 6.14, Part 2, implies that j € block(k, b — 2failbd, aug-ring(a)). Therefore, 
k € block(j,b — 2failbd, aug-ring (a)). 

Then we claim that k performs a neighbor-refresh sometime in the interval \ttime (a) — (2T g +2d) , ttime (a) — 
(T g + 2d)). Let a 1 be the prefix of a ending just before this neighbor-refresh k , let s' = lstate(a'), and 
let R 1 = global-ring (a 1 ). 

Since j € block(k,b — 2failbd, aug-ring(a)), Lemma 6.19 implies that j G block(k,b — 2failbd,R'), 
and so by Lemma 6.16, j € block(k, b — failbd, aug-ring(a')). Also, since k € block(i, c — 2failbd, R), 
we have that i € block(k, c — 2failbd, R), so by Lemma 6.16, i € block(k, c — failbd, aug-ring(a)), so 
by Lemma 6.19, i € block(k, c — failbd, R 1 ), so again by Lemma 6.16, i € block(k, c, aug-ring(a')), so 
i £ block(k,b —failbd, aug-ring (a!)). 

Then by inductive hypothesis, Part 5, s 1 .fingers k contains a finger for each of i and j, both with exptime > 
s' .now + T e — (T g + d). Since j € block(k,b, aug-ring(a')) and s 1 .fingers k contains a finger for 
j, Lemma 6.17 implies that j € block(k,b,s' .local-ring k ). Therefore, j is among the targets of the 
block message sent by k during the neighbor-refresh k . Also, since i € block(k,b, aug-ring(a')) and 
s' .fingers k contains a finger for i, Lemma 6.17 implies that i € block(k, b, s 1 .local-ring k ) . Therefore, the 
finger for i is included in the block sent by k in its block message to j. When the finger is sent, it has 
exptime > s 1 .now + T e — (T g + d). 

This block message arrives at j at a time < ltime(a) — (T g +d) . Then sometime in the interval [ltime(a) — 
(T g + d),ttime(a) — T g ), j performs a neighbor-refresh^. Let a" be the prefix of a ending just before 
this neighbor-refresh j, let s" = £state(a"), and let R" = global-ring(a"). 

Sincej € block(i, b— failbd, aug-ring(a)), we have, by Lemma6.19, thati € block(j,b— failbd, global- ring (a")). 
So by Lemma 6.16, i G block(j, b, aug-ring (a 11 )). Also, s" .fingers ,■ contains a finger for i, because the 

25 



finger for j that arrives in the block message from k has not had time to expire. Then Lemma 6.17 implies 

that* € block(j,b, s" .local-ring A. Therefore, i is among the targets of the block message sent by j during 

this neighbor-refresh . 

This block message contains a finger for j, with exptime = s" .now + T e . Therefore, at the end of a, 

at most time T g + d later, s.fingers i contains a finger for j with exptime > s.now + T e — (T g + d), as 

needed. 

(b) t > Itime(a) - (2T g + 3d). 

Then the time between the join-ack^ and join-ack is > T g + 3d. 

Lemma 6. 15 implies that there exists A; € PId such that k € predset(j,c—2failbd,R)—predset(j,2failbd,R), 

join-ackj. occurs at a time < ltime(a) — (6T g + 13<i), and fail/t occurs in a. This uses the assumption that 

c > Ijoinbd + 4failbd. 

Now we claim that i G block(k,b — 3failbd, aug-ring(a)). Since k € predset(j,c — 2failbd,R), 

Lemma 6.16 implies that k € predset(j, c — failbd, aug-ring(a)). Since k $. predset(j, 2failbd,R), we 

have that A; ^ predset(j,2failbd, aug-ring(a)). Since j € block (i,b— failbd, aug-ring(a)), Lemma 6.14, 

Part 2, implies that i € block(k, b — 3failbd, aug-ring(a)), as claimed. 

Process j performs a join-ping at some time in the interval [t — (T g +2d),t — T g ), and j receives responses 

for all ping messages generated by that join-ping whose destinations do not fail, strictly before time t. Let 

a 1 be the prefix of a ending just before this join-ping^, s 1 = £state(a'), and R 1 = global- ring (a 1 ). 

We claim that s 1 . fingers • contains a finger for k. Since k € block(j, c— failbd, aug-ring(a)), Lemma 6. 19 

implies that k € block(j, c — failbd, R 1 ), and so by Lemma 6.16, k € block(j, c, aug-ring(a')). Then by 

inductive hypothesis, Parts 1 and 2, s' .fingers ,■ contains a finger for k. 

Since k € block(j,c, aug-ring(a')) and s 1 . fingers • contains a finger for k, Lemma 6.17 implies that 

k € block(j, c, s 1 .local-ring A. Therefore, during the join-ping, j sends a ping message to k. Since k does 

not fail, it responds with a block message. Let a" be the prefix of a ending just before k sends this block 

message, let s" = £state(a"), and R" = global-ring(a")). 

Since i € block(k,b — 2failbd, aug-ring(a)), Lemma 6.19 implies that i € block(k,b — 2failbd,R"). 

Then Lemma 6.16 implies that i € block(k, b— failbd, aug-ring(a")). Then by inductive hypothesis, Part 

5, we know that s" .fingers k contains a finger for i with exptime > now + T e — (T g + d). 

Since i € block(k,b, aug-ring(a")) and s" .fingers k contains a finger for i, Lemma 6.17 implies that 

i € block(k, b, s" .local-ring k ) . Therefore, the finger for i is included in the block sent by k in its block 

message to j. This finger is recorded by j, and persists until the end of a. 

Immediately after the join-ack , and at intervals of T g thereafter, process j performs a neighbor-refresh , 

in which it sends a block message containing a finger for itself with exptime = T e . 

We claim that i is included in the set of targets of each such block message. This is because i € block(j, b — 

failbd, aug-ring(a)), so by Lemma 6.16, i € block(j, b, aug-ring) at each point after the join-ack^. Then 

Lemma 6.17 implies that i € block(j, b, local-ring A at each point after the join-ack-, which implies that i 

is included in the set of targets of each such block message. 

Some such message must arrive at i that is sent by j at a time > ltime(a) — (T g + d). Therefore, s. fingers i 

contains a finger for j with exptime > s.now + T e — (T g + d), as needed. 

□ 

6.5 Maintaining the Chords 

We state a lemma analogous to the main lemma of the previous section, Lemma 6.20, but for neighbors of each 
particular chord position x rather than neighbors of the node i itself. 

The statements of Part 1, 2, and 3 are entirely analogous to those in Lemma 6.20. However, in Part 4, the fact 
that i uses chord-pings instead of neighbor-refreshes to keep up-to-date with respect to x after the join-ack^ changes 
the bound slightly. Part 5, which describes situations where i obtains first-hand knowledge of j directly from j, gets 
weakened considerably. This is because we have no phenomenon analogous to that of the prior case 5(b), where j 
informs i directly about its existence immediately after the join. So, the new Part 5 talks only about those j that are so 
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close to the chord position that i pings j directly during its chord-pings. Since i pings only the apparent c-block of x, 
this involves only those j that are in this tiny neighborhood. 

The proof is also different in some interesting ways. Rather than relying on the inductive hypotheses as before, we 
rely on the earlier lemma about neighborhoods, Lemma 6.20. That is because the relevant information arrives from 
neighbors of the chord position x. 

Lemma 6.21. Let a be a good finite execution, s = £state(a). Let i be a process that does not fail in a. Let 
k 6 N, < k < n - 1, and x = PToX(i) + 2 k . Then: 

1. Suppose that s.statusi = joining and a receive(lookup-comp)*j event for target x occurs in a at a time 
> Itime(a) — (T g + 2d). Suppose that j € block(x,c, aug-ring(a)), join-ack^ occurs in a at a time < 
£time(a) — (3T g + 8d), andfaWj does not occur in a. 

Then fingers i contains a finger for j with exptime > now. 

2. Suppose that statusi = joining and a receive(lookup-comp)*j event for target x occurs in a at a time < 
£time(a) — (T g + 2d). Suppose that j € block(x,b, aug-ring(a)), join-ack^ occurs in a at a time < Itime(a) — 
(T g + 3d), and failj does not occur in a. 

Then fingers i contains a finger for j with exptime > now +T e — (2T g + 3d). 

3. Suppose that s.statusi = active and a join-ack^ occurs in a at a time > £time(a) — (T g + 2d). Suppose that 
j € block(x, b, aug-ring(a)), join-ack^ occurs in a at a time < Itime(a) — (2T g + 5d), and failj does not 
occur in a. 

Then s.fingers i contains a finger for j with exptime > s.now +T e — (3T g + 6d). 

4. Suppose that s.statusi = active and a join-ack^ occurs in a at a time < £time(a) — (T g + 2d). Suppose that 
j € block(x, b, aug-ring(a)), join-ack^ occurs in a at a time < £time(a) — (T g + 3d), and f a i I _,- does not occur 
in a. 

Then s.fingers i contains a finger for j with exptime > s.now +T e — (2T g + 3d). 

5. Suppose that s.statusi = active and a join-ack^ occurs in a at a time < £time(a) — (2T g + Ad). Suppose that 
j € block (x, c — failbd, aug-ring(a)), join-ack^ occurs in a at a time < Itime(a) — (2T g +5d), andfaWj does 
not occur in a. 

Then s.fingers i contains a finger for j with exptime > s.now +T e — (T g + 2d). 

Proof. Parts 1, 2, and 3, are proved similarly to before, but instead of inductive hypotheses, they use the relevant 
parts of Lemma 6.20. 

For Part 4, we rely on the chord-ping mechanism. And again, the relevant parts of Lemma 6.20 rather than 
inductive hypotheses. 

For Part 5, we use Part 4 to conclude that i learns about j by time £time(a) — (T g + 2d), and then rely on the 
chord-ping mechanism. The key is that in this last chord-ping, i communicates directly with (pings) j. 

D 

6.6 Correctness of Lookup Results 

Theorem 6.22. Every good execution a satisfies 2T g + (id-lookup-correctness. 

Proof. (Sketch:) Let a' be a prefix of a ending just before a lookup-ack^), event, which is a response to a prior 
lookup(a;)j. Lets' = £state(a') and R 1 = global-ring(a'). 

It suffices to produce a ring R such that R C aug-ring(a'), R contains every Xld in R 1 except possibly for those 
j such that join-ackj occurs in a 1 at a time > £time(a') — (2T g + 6d), and H = ppredset(x, c, R). 

Define the ring R to be the union S U T, where: 

• S is the set of all PToX(j) e R 1 such that join-ack^ occurs at a time < Uime(a') - (2T g + 6d). 

• T is that set of all Xlds in s 1 .fingers t . 
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We show that R satisfies the three properties. 

The first property is immediate, because all Xlds in s' ' .fingers i are in aug-ring(a'). The second property is also 
immediate, because S C R. For the third property, the code for lookup-ack(i : f)j implies that H = ppredset(x, c,s' .local-ring^. 
We need to show that H = ppredset(x, c, R). Since local-ring i is the set of Xlds in fingers^ and that set is a subset 
of R, it is enough to show that every j € ppredset(x, c, R) is also in s 1 .fingers t . 

So, fix j £ ppredset(x,c,R). If j € T then we are done so assume that j € S. Thus, j € R 1 and join-ack occurs 
at a time < t — (2T g + 6d). Since j € ppredset(x, c, R), we have that j € ppredset(x, c + 3joinbd, R 1 ). 

The lookup-ack(i : f)j event follows the receipt by i of a lookup-comp message, with no intervening time passage. 
Let A; be the sender of this lookup-comp message. Then k sent this message at some time > ltime(a') — d. Let a" be 
the prefix of a ending just before k composed this message, s" = lstate(a"), and R" = global-ring (a"). 

We claim that k € ppredset(x, c + 3joinbd, R"); the argument is like one in Lemma 6.20, Part 1. 

Since j € ppredset(x, c + 3joinbd, R 1 ), it follows that j € ppredset(x, c + 3joinbd + failbd, R"). Since j € 
ppredset(x,c + 3joinbd + failbd, R") and k € ppredset(x,c + 3joinbd,R"), it follows that j € block(k,c + 
3joinbd + failbd, R"). Since b > c + 3joinbd + 2failbd, we have that j € block(k,b — failbd, R"). Therefore, 
j € block(k,b, aug-ring(a")). 

By Lemma 6.20, Parts 3 and 4, s" .fingers k contains a finger for j with exptime > s" .now + T e — (3T g + 6d). 
This finger for j gets included in the block sent by k in the lookup-comp message. After i receives this message, 
fingers i contains a finger for j with exptime > now + T e — (3T g + Id). Then, since T e > 3T g + Id, s' .fingers { 
contains a finger for j. This is what we needed to show. □ 

6.7 Latency Bounds 

6.7.1 Latency of a request 

Theorem 6.23. Suppose that a is a good execution, a 1 a finite prefix of a containing at least 2c + 1 join-ack events. 
Suppose that: 

1. The final step of a 1 is a lookup, step in which i initiates request r, with target x. 

2. No other requests (on behalf of joins, client lookups, or stabilizes) are active at any time > £time(a') — T e . 
Then request r terminates with a receive(lookup-comp) step, at a time that is < £time(a') + 4(log N + \)d. 

Proof. (Sketch:) We first claim that, at any point during the lookup, for any process ^ i in the ring, the known 
predecessors of the target x are "bunched together" in at most two c-blocks in the actual global ring. One of these is 
the block of actual predecessors of x in the ring, and the other may be anywhere else. 

Claim 6.24. At any point in a after a', and for any j ^ i, all processes in ppredset(x,c, local-ring A that have 
not failed lie within two c-blocks of consecutive processes in global-ring: ppredset(x,c, global-ring) and one other 
c-block. 

Proof. Everyone except i keeps only its neighborhood and chord fingers, as specified by the underlying infrastruc- 
ture. These have the needed property. (Two blocks can arise if the target x is in the middle of one of j's blocks.) 

□ 

Claim 6.25. At any point in a after a 1 , and before a acirecewe(lookup-comp)j event, all processes in ppredset(x, c— 
Afailbd, local-ring ,) that have not failed lie within two c-blocks of consecutive processes in global-ring: ppredset(x, c, global-ri 
and one other c-block. 

Proof. (Sketch:) This is more complicated than the previous claim, because process i acquires fingers from other 
nodes' tables in the course of the lookup. 

The ways in which process i acquires new fingers are somewhat constrained: by normal neighborhood and chord 
refreshing, by receiving a lookup-resp message or by receiving a lookup-comp message. We rule out the last case by 
assumption — we are considering only what happens before the first receive(lookup-comp)j happens. 

Thus, whenever i acquires new fingers, it acquires an entire block of size at least c from some other node, which 
by the previous claim is included in only two c-blocks in the actual global ring at the time the block was sent, one of 
these blocks being ppredset(x, c, global-ring). 
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Since at most failbd of each of these blocks could have failed before the block was sent, and at most another failbd 
from each of these blocks could fail after the send and up to the point of reference, it must be that at least c — Afailbd 
of the newly-arrived fingers do not fail by the point of reference and lie within two c blocks in global-ring , with one 
of these blocks being ppredset(x, c, global-ring). 

But this doesn't quite tell us that all processes in ppredset(x, c — Afailbd, local-ring t ) that have not failed lie within 
these two c-blocks of consecutive processes in global-ring . For this, we have to use the fact that the blocks in fingers i 
that are closest to x don't "degrade" by having too many processes fail. The reason this doesn't happen is that i keeps 
moving the algorithm along — pinging "enough" nodes among its closest predecessors for x, and receiving responses 
from many of them, which provide information about blocks that are still closer to x. □ 

Now the key claim describes how the "distance" to the destination x is halved every time Ad, until near the end of 
the lookup: 

Claim 6.26. Let ebe a power of two, e < N. 

Suppose that, at some point during the lookup, the clockwise distance from pred(x, c — Afailbd, local-ring J to x (in 

the identifier space) is < e. 

Then by time Ad later, at least one of the following holds: 

1. The lookup ends (with the receipt of a lookup-comp message). 

2. fingers i contains at least c — Ifailbd of the members of ppredset(x, c, global-ring). 

3. The clockwise distance from pred(x, c — Afailbd, local-ring t ) to x is < e/2. 

Proof, (of Claim:) Assume that the lookup doesn't end within time Ad, that is, Case 1 doesn't hold. Then within 
time 2d, process i performs a new join-ping, which results, within an additional time 2d, in a response from one of the 
processes corresponding to the Xlds in the assumed ppredset(x, c — Afailbd, local-ring i ). (The fact that one responds 
depends on the fact that not all of these processes can have failed recently or fail during the ping-response exchange. 
This in turn relies on our assumed bound on failure rate, and the assumption that they are all within two c-blocks in 
the global ring.) 

Let j be such a responding process. If PToX(j) + 1 = x, that is, x is the immediate successor of j in the Xld 
space, then j sends a lookup-comp message, contradicting the fact that Case 1 doesn't hold. So, we may assume that 
x is not the immediate successor of j in the Xld space. 

Then choose k to be the largest natural number such that PToX(j) + 2 k G (PToX(j),x), that is, the largest 
power-of-two successor of j that does not reach x. 

The response from j to i contains a set F of fingers representing j's c best predecessors for x at the time j sends 
its response. There are two cases: 

1. F contains only elements in the open interval (PToX(j) + 2 k ,x). That is, only elements after the given largest 
power-of-two successor of j. 

In this case, after i receives the message, the clockwise distance from pred(x, c — 2failbd, local-ring^ to x is 
< e/2, which suffices to satisfy Case 3. 

2. F contains at least one element that is not in the open interval (PToX(j) + 2 k ,x). 

Lemma 6.21 implies that, when j sends the lookup-response message, fingers j contains entries for all elements 
of block(j + 2 k ,b, augmented-ring) that have not failed. Since the set F contains at least one element that 
is not in the open interval (j + 2 k ,x), we claim that F contains actual predecessors of x in the global ring, 
specifically, F contains at least c — failbd of the members of ppredset(x, c, global-ring) at the time j sends the 
message. (Up to failbd of the fingers in F could have already failed at the time of the send.) Just after i receives 
the message, fingers i contains at least c — 2failbd of the members of ppredset(x, c, global-ring) . This yields 
Case 2. 

□ 

To complete the proof, we use the last claim repeatedly, as long as Case 3 holds. Since we cannot keep halving 
forever, eventually, either Case 1 or Case 2 arises. If Case 1 arises first, then we are done. On the other hand, if Case 
2 arises first, then within only one more ping round, i receives a lookup-comp message, so again we are done. 
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7 Appendix B: Using Nondeterministic Assumptions 

As described in Section 3) and Appendix B our analysis in this paper is based on deterministic assumptions. In general, 
we assume that there are at most v relevant events that occur in an "arc" of the ring containing at most r processes 
during a time interval A. 

These assumptions however are not realistic for many distributed environments. In practice join and failure events 
are modeled by probability distribution functions (e.g, Poisson) which makes it impossible to put a deterministic bound 
of the number of such events during an interval of time A. 

To establish a relationship between the more realistic probabilistic assumptions and the deterministic assumptions 
next we compute the mean time Tf between two violations of the deterministic bounds under the probabilistic as- 
sumptions. In other words, Tf represents the expected time for which a MultiChord will remain in the quasi-ideal 
state. 

For tractability, we assume a system in which processes join according to a Poisson process with arrival rate A a 
and that the process lifetimes are exponentially distributed with a mean of I, Assuming that the MultiChord ring is in 
steady state we have I = N/X a , i.e., the rate of joins is equal to the rate of failures or leaves. Thus, the rate of changes 
is A = 2A a . 

Next, we bound the probability that the deterministic assumption-that no more than v relevant events occur during 
a time interval A in an arc of the ring of r processes — is violated. 

The average number of events that occur in a given arc of the ring consisting of r processes during an interval of 
time A is 

fi = A-\(r/N), (2) 

where A • A represents the average number of events that occur in the entire system during a time interval A, and 
(r/N) represents the fraction of these events that occur during that portion of the ring. 

Because events are generated from a Poisson distribution we can apply the Chernoff bound: 

Pr(X>(l + S)n) <e-^ 2 '\ (3) 

where Pr(X > (1 + S)/i) represents the probability that no more than (1 + 5)/i events occur in a given arc of r 
processes during a time interval A. Taking v = (1 + S)fi, the probability that the deterministic bound is violated in a 
given arc of r processors during a time interval A is 

Pr{X > v) < e 4 f . (4) 

The probability p(A,r) that the deterministic bound is violated in any arc of r processors during an interval A is 
bounded above by 

p(A,r) < NPr(X > v) < Ne~ iJ ^~ . (5) 

Then the mean time Tf between two violations of the deterministic bound is 



Expanding /x yields 



A A (*-») 2 

p(A,r) N 



T f >-e^^. (7) 



where A = X/N represents the normalized rate of change. 

Next, let us consider how do deterministic constraints presented in Section 3 map to Ineq. (7). In particular, we 
consider the following constraints: 
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T e > 5Tj (8) 

c > 7 joinbd + Afailbd 
b > 2c + 3 joinbd + max(2 j oinbd, failbd), (9) 

where failbd represents the number of failures in an arc of b + 1 processes during time T e , and joinbd represents the 
number of joins in an arc of b + 1 processes during time Tj. 

Because we assume steady state, the number of failures and joins in an arc of b + 1 processes is roughly the same 
during a given time interval. This means that failbd = joinbdT e /Tj. If we take T e /Tj = 5, the last two constraints 
in Ineqs. (8) become: 

c > 27joinbd (10) 

b > 61 joinbd 



during an interval of time T e , and 



c > hAjoinbd (11) 

b > 12. 2 joinbd 



during an interval of time Tj . 

Since constraints (11) imply constraints (10) next we consider only constraints (11). Let us take c = Qjoinbd, 
b = 13joind, values which satisfy both these constraints. 

Finally, we take r = b + 1, A = Tj, and v = 2joinbd (the factor of 2 is because v accounts for both joins and 
failures during the interval Tj). With these values, the expected time before the deterministic constraints are violated 
(see Ineq (7)) becomes 



rp (c/3-AT 3 -(6+l)) J 

T f >^-e ^ T i("+ r > , (12) 

1 N 



where b > 13c/6. 
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